3rd International Conference on Information and Communication Technologies and Development

April 17-19, 2009 Carnegie Mellon University in Qatar Education City, Doha, Qatar

Host Organization:

Organizing Partner:

Platinum Sponsors:

Gold Sponsors:

Bronze Sponsors:

Media Partner:

Technical Sponsors:

Table of Contents

Introduction A Global Empirical Evaluation of New Communication Technology Use and Democratic Tendency Victoria Stodden and Patrick Meier A Review of the Research on Mobile Use by Micro and Small Enterprises (MSEs) Jonathan Donner and Marcela Escobari An Evaluation of the Use of ICT within Primary Education in Malawi David Hollow and Paola Masperi Claim Mobile: Engaging Conflicting Stakeholder Requirements in Healthcare in Uganda Melissa R. Ho, Emmanuel K. Owusu and Paul Aoki Computer Games in the Developing World: The Value of NonInstrumental Engagement with ICTs, or Taking Play Seriously Beth E. Kolko and Cynthis Putnam Content Creation and Dissemination by-and-for Users in Rural Areas Sheetal K. Agarwal, Arun Kumar, Amit Anil Nanavati, Nitendra Rajput E for Express: “Seeing” the Indian State through ICTD Renee Kuriyan and Isha Ray Evaluating the Accuracy of Data collection on Mobile Phones: A Study of Forms, SMS, and Voice Somani Patnaik, Emma Brunskill and William Thies FOLKSOMAPS – Towards Community Driven Intelligent Maps for Developing Regions Arun Kumar, Dipanjan Chakraborty, Himanshu Chauhan HIV Health Information Access using Spoken Dialogue Systems: Touchtone vs. Speech Aditi Sharma Grover, Madelaine Plauché, Etienne Barnard, Christiaan Kuun ICT4What? – Using the Choice Framework to Operationalize the Capability Approach to Development Dorothea Kleine

1 3











ICTD for Healthcare in Ghana: Two Parallel Case Studies Rowena Luk, Matei Zaharia, Melissa Ho, Brian Levine and Paul M. Aoki Improving Child Literacy in Africa: Experiments with an Automated Reading Tutor G. Ayorkor Mills-Tettey, Jack Mostow, M. Bernadine Dias, Tracy Morrison Sweet, Sarah M. Belousov, M. Frederick Dias, Haijun Gong Improving Literacy in Rural India: Cellphone Games in an After-School Program Matthew Kim, Anuj Kumar, Shirley Jain, Akhil Mathur, and John Canny Kelsa+: Digital Literacy for Low-Income Office Workers Aishwarya Lakshmi Ratan, Sambit Satpathy, Lilian Zia, Kentaro Toyama, Sean Blagsvedt, Udai Singh Pawar, Thanuja Subramaniam Mapping the Dynamics of Social Enterprises and ICTD in Cambodia Kelly Hutchinson and Alemayehu Molla Political Incentives and Policy Outcomes: Who Benefits from TechnologyEnabled Service Centers? Jennifer Bussell Results from a Study of Impact of E-government Projects in India Subhash C. Bhatnagar and Nupur Singh The Contribution of User-Based Subsidies to the Impact and Sustainability of Telecenters – the eCenter Project in Kyrgyzstan Michael L. Best, Dhanaraj Thakur and Beth E. Kolko









Poster Papers A Speech Enabled Indian Language Text-to-Braille Transliteration System Tirthankar Dasgupta and Anupam Basu Analyzing Statistical Relationships between Global Indicators through Visualization Prabath Gunawardane, Erin Middleton, Suresh Lodha, Ben Crow and James Davis ATMosphere: A System for ATM Microdeposit Services in Rural Contexts Michael Paik and Lakshminarayanan Subramanian

201 201



Building a Transportation Information System Using Only GPS and Basic SMS Infrastructure Ruth E. Anderson, Anthony Poon, Caitlin Lustig, Waylon Brunette, Gaetano Borriello, Beth E. Kolko Challenges in Health Information Systems Integration: Zanzibar Experience Edwin Nyella Cross Technology Comparison for Information Services in Rural Bangladesh Faheem Hussain and Rahul Tongia Decentralization, Clientelism and Popular Participation - Is There a Role for ICTs to Improve Local Governance? Björn-Sören Gigler Design and Deployment of a Blood Safety Monitoring Tool S. Thomas, A. Asuntogun, J. Pitman, B. Mulenga, and S. Vempala Dimensions of IT Literacy in an Arab Region Study in Barkha(Oman) Sherif M. Aziz Emergency Communication and System Design: The Case of Indian Ocean Tsunami R. Chen, J. Coles, J. Lee, and F. R. Rao Empowering Muslim Youth through Computer Education, Access, Use: A Gender Analysis Farida Khan and Rehanan Ghadially eServices Provisioning in a Community Development Context through a JADE MAS Platform Mamello Thinyane, Alfredo Terzoli, Peter Clayton Extending the Technology-Community-Management Model to Disaster Recovery in Asia Arul Chib and A.L.E. Komathi Featherweight Multimedia for Information Dissemination Gerry Chu, Sambit Satpathy, Kentaro Toyama, Rikin Gandhi, Ravin Balakrishnan, S. Raghu Menon ICT Governance in Higher Education: Case Study of the Rise and Fall of Open Source in a gulf University Sofiane M. Sahyraoui













ICTD State of the Union: Where Have We Reached and Where Are We Headed? Rabin Patra, Joyojeet Pal, Sergiu Nedevschi Information Communication Techology and Sustainable Communities in Africa: The Case of the Niger Delta Region of Nigeria (Feb. 2009) Uduak A. Okon Integrating Health Information Systems in Sierra Leone Johan Sæbø, Edem Kwame Kossi, Romain Tohouri Golly-Kobrissa, Ola Titlestad, Jørn Braa Mobile Telephony Access and Usage in Africa A. Chabossou, C. Stork, M. stork, Z. Zahonogo Numeric Paper Forms for NGOs Gusharan Singh, Leah Findlater, Kentaro Toyama, Scott Helmer, Rikin Gandhi, Ravin Balakrishnan Rajnikant’s Laptop: Computers and Development in Popular Indian Cinema Joyojeet Pal Regulatory Independence and Wireless Market Development: A Comparative Analysis of Two African Nations Annemijn F. van Gorp and Carleen F. Maitland Social Enterprises: A Vocational Entrepreneurship Framework for Street Youth Paul Javid, Kentaro Toyama, Manna Biswas Speech vs. Touch-Tone: Telephony Interfaces for Information Access by Low Literate Users Jahanzeb Sherwani, Sooraj Palijo, Sarwat Mirza, Tanveer Ahmed, Nosheen Ali, Roni Rosenfeld The Case for SmartTrack Michael Paik, Ashlesh Sharma, Arthur Meacham, Giulio Quarta, Philip Trahanas, Brian Levine, Mary Ann Hopkins, Barbara Rapchak, Lakshminarayanan Subramanian Uses of Mobile Phones in Post-Conflict Liberia Michael L. Best, Edem Wornyo, Thomas N. Smyth and John Etherton














An Automated Braille Writing Tutor with Multilingual Exercises and Educational Games M. Bernadine Dias, M. Freddie Dias, Sarah Belousov, Mohammed Kaleemur Rahman, Saurabh Sanghvi, Imran Fanaswala, Wael Ghazzawi, Ameer Abdulsalam, Noura El-Moughny, and S. Raghu Menon Boosting European Market Access to Malian Mango Growers Saskia Harmen Creating a Mobile-Phone Based Geographic Surveillance System for Asian Flu Yibo Lin and Claire Heffernan Design of a Blood Flow System A. Osuntogun, S. Thomas, J. Pitman, S. Basavaraju, B. Mulenga and S. Vempala DISHA: DISease and Health Awareness for Children on Multiple Input Devices Mohit Jain, Aakar Gupta, Navkar Samdaria, Praveen Shekhar and Joyojeet Pal Freedom Fone: Dial-up Information Service Bev Clark and Brenda Burrell FrontlineSMS and Ushahidi – A Demo Ken Banks and Erik Hersman Global Youth Connectivity (GYC) – Using ICT for Peaceful Recovery and Long-term Change Anne Bertrand Implementing E-Government Accessible to Illiterate Citizens D. Kettani and A. El Mahdi Improving Data Quality with Dynamic Forms Kuang Chen, Harr Chen, Neil Conway, Heather Dolan, Joseph M. Hellerstein, and Tapan S. Parikh IWB4D – Interactive Whiteboards for Development John Traxler and Lee Griffiths












Livestock, Learning and Diagnostics: New Directions in Veterinary Tele-medicine Jun Yu and Claire Heffernan Metamouse: Multiple Mice for Legacy Applications Kurtis Heimerl, Divya Ramachandran, Joyojeet Pal, Eric Brewer, and Tapan Parikh Mobile Phone Job Services: Linking Developing-country Youth with Employers, via SMS Amber Houssain, Mohammad Kilany, and Jacob Korenblum MultiMath: Numeric Keypads for Math Learning on Shared Personal Computers Sunil Garg, Charlotte Robinson, Clint Tseng, Heather Underwood, Richard Anderson, Joyojeet Pal A New Generation of Open Source Data Collection Tools Yaw Anokwa, Carl Hartung, Adam Lerer, Brian DeRenzi, Gaetano Borriello RuralScope: An Information System for Tracking Rural Disbursements Sai Gopal Thota, Rabin Ratra, Murali Medisetty, Sivananda Reddy, Vivek Mungala, Joyojeet Pal T-Cube Web Interface in support of Real-Time Bio-surveillance Program Artur Bubrawski, Maheshkumar Sabhnani, Michael Knight, Michael Baysek, Daniel Neill, Saswati Ray, Anna Michalska and Nuwan Waidyanatha Web Search over Low Bandwidth Jay Chen, Lakshminarayanan Subramanian, and Jinyang Li Author Index










ICTD 2009 – Organizing Committee
Honorary Chairs
Dr. Hessa Sultan Al-Jaber Secretary General ictQATAR Dr. Kentaro Toyama Assistant Managing Director Microsoft Research India (MSRI)

Conference Chair
Dr. M. Bernardine Dias Carnegie Mellon University

Program Committee Chairs
Dr. Richard Heeks University of Manchester Dr. Rahul Tongia Carnegie Mellon University

Advisory Board
Dr. Vallampadugai S. Arunachalam Center for Study of Science, Technology, and Policy (CSTEP), Bangalore Dr. Michael Best Georgia Institute of Technology Dr. Balaji Parthasarathy International Institute of Information Technology (IIIT), Bangalore Dr. Raj Reddy Carnegie Mellon University Dr. Kentaro Toyama Microsoft Research India Dr. Francois Bar University of Southern California

Dr. Kenneth Keniston Massachusetts Institute of Technology Dr. Krithi Ramamritham Indian Institute of Technology (IIT), Bombay Dr. AnnaLee Saxenian University of California, Berkeley Dr. Ernest Wilson University of Southern California

Senior Program Committee Members
Dr. Francois Bar University of Southern California Dr. Eric Brewer University of California, Berkeley Dr. Michael Best Georgia Institute of Technology Dr. Chris Coward University of Washington

Dr. Robert Davidson City University of Hong Kong Dr. Shirin Madon London School of Economics Dr. Balaji Parthasarathy International Institute of Information Technology (IIIT)-Bangalore Dr. Kentaro Toyama Microsoft Research India

Dr. Hernan Galperin Universidad de San Andrés Dr. Alemayehu Molla Royal Melbourne Institute of Technology Dr. Krithi Ramamritham Indian Institute of Technology (IIT)-Bombay

Dr. Tim Unwin Royal Holloway University of London

Panels and Workshops Chairs
Dr. Joseph Mertz Carnegie Mellon University Dr. Joyojeet Pal University of Washington

Publications Chairs
Dr. Yonina Cooper Carnegie Mellon University in Qatar Dr. Thrishantha Nanayakkara Harvard University and University of Moratuwa, Sri Lanka

Poster Chairs
Dr. Faheem Hussain Carnegie Mellon University in Qatar Dr. Tapan Parikh University of California, Berkeley

Demo Chairs
Dr. Khaled Harras Carnegie Mellon University in Qatar Bill Thies Massachusetts Institute of Technology

Regional Chairs

Dr. Gary Marsden University of Capetown, South Africa Dr. Tim Waema University of Nairobi, Kenya

Middle East
Dr. Adnan Abu Dayya Qatar University Dr. Fouad Mrad American University of Beirut

East Asia
Dr. Jack Linchuan Qiu The Chinese University of Hong Kong Rinalia Abdul Rahim Global Knowledge Partnership, Malaysia

South Asia
Anita Gurumurthy IT for Change, India Dr. Umar Saif Lahore University of Management Sciences, Pakistan

South and Central America
Dr. Nicolau Reinhard Universidade de São Paulo, Brazil Dr. Osvaldo Rodriguez La Planta University, Argentina

North America
Dr. John Bennett University of Colorado at Boulder Claudia Morrell Multinational Development of Women in Technology

John Traxler University of Wolverhampton, UK

Christina Higa University of Hawaii Dr. Esther Batiri Williams The University of the South Pacific

Technical Program Committee Members
Dr. Jessica Aalami University of California, Berkeley Dr. Reuben Abraham Indian School of Business, Hyderabad Dr. Richard Anderson University of Washington Akhtar Badshah Microsoft Dr. Anupam Basu Indian Institute of Technology (IIT)-Kharagpur, India Dr. Salam Abdallah Abu Dhabi University Dr. Erwin Alampay University of Philippines Dr. Peng HwaAng Nanyang Technological University Dr. V. Balaji ICRISAT Dr. John Bennett University of Colorado, Boulder

Dr. Subhash Bhatnagar Indian Institute of Management, Ahmedabad Dr. John Canny University of California, Berkeley

Robert Bichler University of Salzburg

Dr. Royal Colle Cornell University

Dr. Rahul De Dr. Andy Dearden Indian Institute of Management, Bangalore Sheffield Hallam University Dr. Uday Desai Indian Institute of Technology (IIT)-Bombay Dr. Jonathan Donner Microsoft Research Dr. Kevin Fall Intel/Berkeley Dr. Alison Gillwald University of Witwatersrand Pat Hall Kathmandu University Dr. Antonio Diaz University of Auckland

Dr. Ayman Elnaggar Sultan Qaboos University Dr. Ping Gao University of Manchester Dr. Gillian Green University of Bolton Dr. Saskia Harmsen International Institute for Communication and Development (IICD) Dr. Bill Hefley Carnegie Mellon University Dr. Faheem Hussain Carnegie Mellon University in Qatar Dr. Ashok Jhunjhunwala Indian Institute of Technology (IIT)-Madras Matt Kam University of California, Berkeley Dr. Atreyi Kankanhalli National University of Singapore Dr. G. R. Kiran London School of Economics

Dr. Claire Heffernan University of Reading Dr. Heather Hudson University of San Francisco Mahad Ibrahim University of California, Berkeley

Dr. Muhammadou M. O. Kah American University of Nigeria Dr. Sherif Kamel American University of Cairo Dr. Srinivasan Keshav University of Waterloo

Dr. Dorothea Kleine University of London Dr. Beth Kolko University of Washington Dr. Gillian Marcelle WITS University Dr. Wagner Meira UFMG – Federal University at Minas Gerais Dr. Harekrishna Misra Institute of Rural Management, Anand Dr. Beda Mutagahywa University of Dar es Salaam Dr. Amit Nanavati IBM Dr. Joyojeet Pal University of Washington Dr. Fay Payton North Carolina State University

Dr. Jim Koch Santa Clara University Richa Kumar Massachusetts Institute of Technology Dr. Victor Mbarika Southern University and A&M College Dr. Michel Menou City University of London

Dr. Amit Mitra Cranfield University Dr. Shrikant Naidu Motorola Labs, India Dr. Solomon Negash Kennesaw State University Dr. Tapan Parikh University of California, Berkeley Francisco Proenza Food and Agriculture Organization of the United Nations (FAO)

Dr. Ranjini Raghavendra Dr. Nicolau Reinhard Indian Institute of Management, Bangalore University of Sao Paolo Marijn Rijken TNO (Netherlands Organisation for Applied Scientific Research) Dr. Roni Rosenfeld Carnegie Mellon University Dr. Tony Salvador Intel Dr. Afzal Sher Swedish Program on ICT in Developing Regions (SPIDER) Dr. Osvaldo Rodriguez La Planta University

Dr. Sofiane Sahraoui American University of Sharjah Dr. Maung Sein University of Agder Jahanzeb Sherwani Carnegie Mellon University

Dr. Nirvikar Singh University of California, Santa Cruz Dr. Christoph Stork Wits University

Dr. Hettie Soriyan Obafemi Awolowo University Dr. Eswaran Subrahmanian CMU/NIST/CSTEP

Dr. Lakshmi(narayanan) Subramanian Bill Thies Courant Institute of Mathematical Sciences Massachusetts Institute of Technology Dr. Cathy Urquhart University of Auckland Dr. Timothy Waema University of Nairobi Dr. Adel El Zaim International Development Research Centre (IDRC) Canada Dr. Carinade Villiers University of Pretoria Dr. Ernest Wilson University of Southern California

Local Organizing Committee Chairs
Murry Evans Carnegie Mellon University in Qatar Erin Stewart Carnegie Mellon University in Qatar Elaine Farah ictQatar

Publicity Coordinators
Noha Al Afifi Carnegie Mellon University in Qatar Andrea Zrimsek Carnegie Mellon University in Qatar

Logistics Coordinators
Renee Barcelona Carnegie Mellon University in Qatar Kara Nesimiuk Carnegie Mellon University in Qatar Sarah Belousov Carnegie Mellon University Ermine Teves Carnegie Mellon University

Website Coordinators
M. Freddie Dias Carnegie Mellon University Daniel Freeman Carnegie Mellon University

Local Organizing Committee Members
Ray Corcoran, Bob Gaus, Jim Gartner, Shamila Khader, Aaron Lyvers Carnegie Mellon University in Qatar


Introduction to the Proceedings of ICTD2009
Welcome to the 3rd International Conference on Information and Communication Technologies and Development (ICTD2009). It is with great pleasure that we present the ICTD2009 proceedings, which include all of the full papers presented at the conference in Doha, Qatar, held on 17-19 April 2009. ICTD is the premier series of scholarly conferences on the use of ICTs for development, spanning technical and social science domains. The call for papers attracted a record 250 submissions. All papers were put through a double-blind peerreview process. The Program Co-Chairs assigned papers to our Senior Program Committee members who oversaw a review process involving three Program Committee reviewers per paper: one with deep expertise about the subject matter; another with broad background in the area; and one drawn from an altogether different discipline. Our continuing hope is that this encourages a convergence of vocabulary and ideas within the ICTD field, while maintaining the integrity of different disciplines. The Senior Program Committee members then meta-reviewed the papers. Authors were allowed a brief rebuttal to reviewer comments before final acceptance decisions were made and revisions were finalized. Ultimately, 19 papers were selected for oral presentation, and another 27 papers were chosen as full papers for poster presentation; an acceptance rate of just over 18%. These papers represent some of the best work being done in ICTD today. They focus on a wide variety of development goals, and involve a broad and innovative range of digital technologies. They draw from all continents of the global South, and focus on all stages of the ICTD lifecycle: from readiness through design and adoption to use and impact. They also tell us about all levels, from the individual through communities and projects to ICTD programmes and policies. We hope that you will find them an insightful, provocative, and informative contribution to our fast-growing field of research and practice. We wish to thank our keynote speakers William H. Gates, Chairman of Microsoft Corporation and Co-Chairman of the Bill and Melinda Gates Foundation, and Dr. Carlos A. Primo Braga, Director of Economic Policy and Debt in the Poverty Reduction and Economic Management Network (PREM) at The World Bank for their insightful presentations. All the other conference presenters also deserve our gratitude for the variety of content and insights they added to the conference program. We also need to thank a number of people without whom the program could not have been put together. First, we wish to thank our Honorary Chairs, Dr. Hessa Sultan AlJaber, Secretary General of The Supreme Council of Information and Communication Technology (ictQatar), and Dr. Kentaro Toyama, Assistant Managing Director of Microsoft Research India (MSRI) for their tremendous support in making this conference a success. We are of course deeply indebted to our Program Committee: those who did the hard work of reviewing and our senior PC members who managed the review process so effectively; all together it is they who ultimately steer the course of ICTD. We thank the Regional Chairs for their assistance in promoting the conference, and our Advisory Board which provided guidance and moral support. We also thank the Publication Chairs, Yonina Cooper and Thrishantha Nanayakkara, who


made this proceedings possible; and Faheem Hussain and Tapan Parikh, who carried the load in organisation of the poster presentations. The conference program for ICTD2009 went well beyond papers, and was significantly enhanced by demonstrations, organised by Bill Thies and Khaled Harras, and by a series of panels and workshops, organised by Joe Mertz and Joyojeet Pal. We are very grateful for their input and hard work. This conference would not have been possible without the tireless efforts of a number of organizers and volunteers, notably our local organizing committee chairs Elaine Farah, Murry Evans, and Erin Stewart, and the many others on the local organizing committee, especially Dean Charles Thorpe, Sarah Belousov, Ermine Teves, Renee Barcelona, Kara Nesimiuk, Andy Zrimsek, Noha Al Afifi, Shams Hassan, Aaron Lyvers, Shamila Khader, Ray Corcoran, Bob Gaus, Freddie Dias, Daniel Freeman, and Jim Gartner. If there are others we did not name explicitly, it is our lapse. The success of the conference is in part due to our many sponsors and partners. We are extremely grateful to our organizing partner, The Supreme Council of Information and Communication Technology (ictQatar), to our media partner Al Jazeera Children’s Channel (JCC), to our technical sponsors IEEE and ACM, and to our financial sponsors, the Qatar National Research Fund (Platinum Sponsor); Canada’s International Development Research Centre (Platinum Sponsor); Qatar Telecom (Platinum Sponsor); ExxonMobil (Gold Sponsor); Microsoft Corporation (Gold Sponsor); IBM (Bronze Sponsor); the Computer Science program at Carnegie Mellon University in Qatar (Bronze Sponsor); and other sponsors who asked not to be publicly acknowledged. Finally, we are indebted to Carnegie Mellon University in Qatar for hosting the conference at their campus in Education City, and to the numerous employees of Carnegie Mellon University (in both the Pittsburgh and Doha campuses), especially the TechBridgeWorld team, who truly made this conference a success. Thanks to everyone who contributed to ICTD2009 in so many ways, and thank you for participating! We are confident you will find the following papers, abstracts, and information not only interesting and useful, but the seeds for further research, innovation, and developmental impact. Richard Heeks, University of Manchester Rahul Tongia, Carnegie Mellon University Program Chairs M. Bernardine Dias, Carnegie Mellon University Conference Chair


A Global Empirical Evaluation of New Communication Technology Use and Democratic Tendency
Victoria Stodden
Berkman Center for Internet and Society Harvard Law School Cambridge, MA 02138 vcs@stanford.edu

Patrick Meier
Berkman Center for Internet and Society Harvard Law School Cambridge, MA 02138 patrick.meier@tufts.edu

Abstract—Is the dramatic increase in Internet use associated with a commensurate rise in democracy? Few previous studies have drawn on multiple perception-based measures of governance to assess the Internet’s effects on the process of democratization. This paper uses perception-based time series data on “Voice & Accountability,” “Political Stability,” and “Rule of Law” to provide insights into democratic tendency. The results of regression analysis suggest that the level of “Voice & Accountability” in a country increases with Internet use, while the level of “Political Stability” decreases with increasing Internet use. Additionally, Internet use was found to increase significantly for countries with increasing levels of “Voice & Accountability.” In contrast, “Rule of Law” was not significantly affected by a country’s level of Internet use. Increasing cell phone use did not seem to affect either “Voice & Accountability,” “Political Stability” or “Rule of Law.” In turn, cell phone use was not affected by any of these three measures of democratic tendency. When limiting our analysis to autocratic regimes, we noted a significant negative effect of Internet and cell phone use on “Political Stability” and found that the “Rule of Law” and “Political Stability” metrics drove ICT adoption. Index terms—cell phone, democracy, fixed effects model, ICT, internet

I. I NTRODUCTION Does the globalization of the Internet have a democratizing effect? The question has already been posed by numerous studies but these have largely taken the form of qualitative case studies and/or large theoretical analyses. In terms of a rigorous, quantitative establishment of the democratization effects of the Internet, however, the jury is still out [1]. At the heart of this debate, moreover, lies a more fundamental question about the essence of democracy. In fact, “unless we are clear about what democracy means to us, and what kind of democracy we envision, technology is as likely to stunt as to enhance the civic polity” [2]. The purpose of this paper is to contribute more rigorous data-driven analysis to the literature on Internet and democracy since “there is no doubt that rigorous and datadriven analysis of this relationship will benefit scholars and policymakers alike” [1]. Previous research on the topic of Internet and democracy can be characterized as lacking a serious perusal of the

democracy and regime transitions literature. To be sure, “the trouble with the zealots of technology as an instrument of democratic liberation is not that they misconceive technology but that they fail to understand democracy” [2]. ‘In other words, “it turns out there is no simple general answer to the question: Is the technology democratizing?’ until we have made clear what sort of democracy we intend.” We address this question first before proceeding with a more detailed literature review. Barber’s notion of “strong democracy” comprises the careful and prudent judgment of citizens who participate in deliberative, self-governing communities. Schmitter and Karl write that, “modern political democracy is a system of governance in which rulers are held accountable for their actions in the public realm by citizens, acting indirectly through the competition and cooperation of their elected representatives”[3]. The two authors emphasize that citizens are the most distinctive element in democracies. “All regimes have rulers and a public realm, but only to the extent that they are democratic do they have citizens” [3]. In contemporary studies of democracy and particularly in pluralist theory, “a vibrant civil society is usually regarded as an essential for good governance and effective democratic consolidation” [4]. In other words, regular elections are not sufficient. As Zakaria noted, illiberal democracies have free elections but citizens remain cut off from real power due to the lack of civil liberties [5]. Huber et al. write that the most basic feature of democracy is power sharing [6]. They identify three clusters of power as primarily relevant for the chances of democracy: (1) the balance of power in civil society; (2) the balance of power between state and society; and (3) the transnational balance of power that shape the first two and constrain political decisionmaking. By remaining diverse and independent of the state, political participation by civil society acts as a channel of public voice and accountability, and a way of challenging and checking the unbridled power of authoritarian regimes [4]. The structure of state-society relations is equally relevant for democracy. As Huber et al. note, “the power of the state needs to be counterbalanced by the organizational strength of


the civil society to make democracy possible; the state must not be so strong and autonomous from all social forces as to overpower civil society and rule without accountability.” Clearly then a governing body that fails to follow the “rule of law,” should not be considered democratic [3]. These elements of democracy are not sufficient conditions for a stable democracy, but they are necessary and indispensable to the persistence of democratic governance. The italicized terms above represent the fundamentals behind the sort of democracy we intend: active citizen participation, good governance, accountability, power sharing, balance of power and rule of law. The few quantitative studies that do exist on Internet and democracy tend to aggregate these fundamentals of democracy into a single index. Doing so means these lose important information on how these individual components of democracy may be affected by the growing prevalence of global Internet access. Furthermore, past quantitative and qualitative studies tend to focus primarily on the impact of the Internet on established democracies. They also focus on the 1990s almost exclusively, a serious limitation that remains surprisingly understated in the literature. Equally problematic in the current literature is the interchangeable use of the terms “Internet” and “information revolution.” The terms are purposefully not differentiated on the basis that the predominant feature of the information society is the spread of the Internet. While this is true of Western democracies, it is certainly not true for the majority of developing, nondemocratic countries, where cell phones are the most widely spread communication technology after radios [7]. Indeed, the irony is that “those who might most benefit from the net’s democratic and informational potential are least likely to have access to it, the tools to gain access, or the educational background to take advantage of the tools” [2]. This paper seeks to redress each of these shortcomings. First, since the boundaries of the term “democracy,” and how it is measured, is subject to lively debate, we use multiple perception-based measures of governance for our dependent variables. Governance indicators provide a better set of proxies for the sort of democracy we intend as identified above. We therefore draw on the following three World Bank indicators: (1) Voice and Accountability (VA) measuring perceptions of the extent to which a country’s citizens are able to participate in selecting their government, as well as freedom of expression, freedom of association, and a free media; (2) Rule of Law (RL) measuring perceptions of the extent to which agents have confidence in and abide by the rules of society, and in particular the quality of contract enforcement, property rights, the police, and the courts, as well as the likelihood of crime and violence; and (3). Political Stability and Absence of Violence (PS) measuring perceptions of the likelihood that the government will be destabilized or overthrown by unconstitutional or violent means, including politically motivated violence and terrorism. These metrics are drawn from the World Bank Governance Indicators Research Database (see http://info.worldbank.org/governance/wgi/). Second, we draw on data from 2000 through 2006, a time

when Internet access and cell phone use is significantly more prevalent and globalized than in the 1990s. If a statistically significant relationship between Internet and democracy does exist, then it is more likely to manifest itself now and not in the 1990s. Third, we draw on both Internet and cell phone data per 100 inhabitants per 181 countries to assess the impact of the information revolution on democratization. We use regression analysis to determine whether Internet or cell phone use has had a correlative effect on measures of democratic tendency. We also model whether the collection of democratic measures has had a correlative effect on Internet or cell phone use. The paper is structured as follows: the first section reviews the current debate and literature on Internet and democracy. The second section explicates the datasets used in this study while the third section formalizes the statistical models employed in the regression analysis. Section four reviews the results and provides an interpretation of the findings. The fifth and final section concludes the study. II. L ITERATURE R EVIEW The Internet and democracy literature comprises two distinct schools of thought each comprising a host of qualitative research and some quantitative inquiry. In this section we review in some detail the qualitative and quantitative literatures that have contributed to both schools of thought over the past ten to fifteen years. In so doing, we compare and contrast the main arguments along with the respective findings. As noted in the introduction, one common shortcoming of the Internet and democracy literature is the tendency to oversimplify our understanding of democracy. The purpose of this literature review is thus to redress this gap evident in previous studies. The first school of thought is often referred to as the more populist school of thought. This strand of the literature subscribes to the viewpoint that the Internet has democratic relevance and impact [8], [9], [10], [11]. According to these authors, the Internet will decentralize access to communication and information while increasing citizen access [12]. Best and Wade write that “the Internet’s collective characteristics (e.g., low cost, multidirectional capability, etc.), helps make this possible.” We first review the qualitative literature that comprises this school of thought followed by quantitative studies. Dahl previously observed that telecommunications technologies have a key role in making possible the advanced democratic country, where policy is firmly anchored in the judgment of the “demos” [13]. In his list of the procedural minimal conditions that must be present for modern political democracy to exist, Dahl thus argues that citizens should have the right to seek out alternative sources of information. Rheingold has dubbed the Internet as “the great equalizer” because it can “equalize the balance of power between citizens and power barons” [14]. The idea here “is that the Internet will serve as a mass audience, and will politicize them in the process” [12]. Anderson et al. draw on sociological research to show that electronic networks lead to the “break-down of status-based social structures” and “increased participation in


discussion, decision-making, and task processes by those who typically are political or economically disadvantaged” [15]. Etzioni sees in the Internet the possibility of an advancement of the state of public affairs through “teledemocracy” [16]. Other scholars claim that the Internet will “enable a Jeffersonian revolution” [17]. Grossman argues that a “third epoch of democracy is arriving by the hand of technology,” and claims that a “new, hybrid electronic republic’ is now forming to displace the creation of Montesquieu, Locke, Madison and their contemporaries” [18], [12]. The practice of “electronic democracy,” according to Browning, will differ substantially from previous renderings of democracy [19]. Perhaps the most provocative claim associated with this more populist school of thought is the one made by Negroponte, who argues that the nation-state will evaporate as a consequence of the information revolution [20]. Snider suggests that citizens need only be potentially informed in order to hold government accountable [21]. Barber opines that by favoring decentralization, the multiplication of choice, and consumer sovereignty, new technologies such as the internet or cell phone have already, albeit inadvertently, benefited democratic political culture [2]. According to Barber, “democracy is a form of government that depends on information and communication. It is obvious then that new technologies of information and communications can be nurturing to democracy. They can challenge passivity, they can enhance information equality, they can overcome sectarianism and prejudice, and they can facilitate participation in deliberative political processes.” Hill and Hughes argue that those who subscribe to the populist school have reasons to be optimistic: “If the mere fact that political discourse against repressive governments is taking place is a good in itself, then the utopians have reason to celebrate [22]. Perhaps the Internet will bring about a wider democratic revolution in the world” [22]. Bimber is more cautious, arguing that while the Internet is accelerating the process of issue group formation and action (in America), the structure of political power has not been revolutionized or qualitatively transformed into a new epoch of democracy [12]. According to Bimber, other scholars believe that the Internet has a “transformative potential” because it facilitates a kind of “one-to-one interaction among citizens and between citizens and government.” Along these lines, some scholars such as Corrado and Firestone write that the Internet has the potential to promote “unmediated” communication and thereby decrease citizens’ reliance on officials and organizations [23]. In sum, what distinguishes the populist enthusiasm for the Internet is the “idea that elites and political intermediaries will grow less important” [12]. In the more contemporary, qualitative literature, Steele and Stein, argue that the Internet amplifies trends in international relations [24]. Rosenau and Johnson address the impact of the Internet at both the individual and international level [25]. At the individual level, the authors argue that the Internet can be used as a tool to organize collectively to effect social and political change around the world (see http://www.DigiActive.org,

for example). At the international level, the authors make the bold claim that the Internet has “contributed to the rise of a more multicentric world structure in which nation-states have seen their preeminence lessen and non-governmental actors take the stage” [1]. The salient point here is that groups and individuals can far more efficiently form coalitions of consequence with a range of powerful collectives. As Best and Wade rightly note, there are obvious democratic elements to this, including the need for “nation-states to provide democratic rights to their citizens so as to build legitimacy on the global stage” [1]. We now turn to the quantitative studies that comprise the first school of thought. One of the earlier statistical studies on this side of the literature was carried out by Kedzie, who provides an account of how information communication technologies contributed to the “third wave” of democracy [26]. Prior to the fall of the Soviet Union and the proliferation of new democracies in Eurasia, the mainstream theory of democratization held that democracy followed economic growth and development [27]. To be sure, one of the few robust findings in the literature is that democracy is more likely in more developed countries [28], [29], [30]. Longdregan and Poole have also shown that the most significant predictor of transitions to authoritarianism is poverty [31], [32]. “In short, after 20 years of observation and analysis during the third wave of academic interest in democratization, we can be reasonably certain that a positive relationship between development and democracy exists, though we do not know why” [33]. Kedzie, however, was more interested in testing another potential causal mechanism, the “dictator’s dilemma” hypothesis, which suggests that the globalization of markets places pressure on authoritarian regimes to keep their countries’ communication borders open. He reasoned that the ensuing massive flow of information would not only allow for “the efficient passage of commercial information, but also for more ’democratic’ information” [1]. As Bimber observes, the most important predictions about the Internet’s impact on politics amount to “causal claims regarding the effect of information flow on political participation and the organization of interests” [12]. Other scholars have made related arguments. Webster, for example, writes that the Internet has helped to facilitate a new form of capitalism called “information capitalism” in which global labor markets require highly flexible workers who continuously adapt and learn [34]. Regimes that impose restrictions on information capitalism forgo the financial returns possible by tapping into the information economy [35], [36], [37]. In his study, Kedzie employs regression analysis to compare how much of the variation in democracy is explained by both traditional predictors of democracy and the strength of Internet diffusion by drawing on data from 144 countries [26]. For his set of control variables, Kedzie included more traditional predictors of democracy including economic development, education, human development and health. He also included indicators of pre-Internet information communication technologies (ICTs). His results suggest that the Internet is indeed


a strong predictor of democracy, more so than traditional determinants of democracy. In a follow up study, Richards assessed the relationship between the Internet and physical integrity [38]. His findings support Kedzie’s. However, the latter study faces an important limitation since Kedzie’s (rather simple) longitudinal analysis draws on data from 1993. At this point during the early 1990s, the Internet was hardly globalized. Best and Wade recognize this important short coming in Kedzie’s study and therefore explore the global effect of the Internet on democracy over a ten year period, 1992−2002 [1]. They aggregate political and civil rights data from Freedom House to formulate a democracy index, which serves as their dependent variable. The number of Internet users per 1, 000 represents their independent variable while the following measures are used as control variables: GDP per capita, education and literacy rates, life expectancy, urbanization, prevalence of non-Internet ICTs. Their analysis shows that a statistically powerful correlation exists between Internet diffusion and level of democratization. “The more salient observation to make, however, is that while economic prevalence and literacy maintain relatively constant correlations with democracy, the correlation for Internet prevalence gradually strengthens, almost to the same level as economic prevalence” [1]. The authors suggest that this dynamic reflects the growing significant relationship between Internet prevalence and democracy: “perhaps this is an indication that the Internet has come of age as a correlate of democracy” [1]. Indeed, they posit that this growth in correlation strength might “be expected given the positive network externalities, the network effect’ that is a salient property of the Internet” [1]. However, the coefficients from the regression analysis reveal that Internet usage is only able to predict a minimal amount of the variation in democracy: “to generate one point of democracy, an extra 500 Internet users per 1,000 citizens is needed, or an extra $5, 882 of GDP per capita is needed” [1]. The scale of democracy runs from 2 to 14. In terms of democracy’s traditional determinants, GDP was a weak predictor while literacy turned out to have no significance whatsoever. The other control variables used were either insignificant or internally correlated. While Best and Wade’s important contribution to the literature on Internet and democracy is one of the few contemporary quantitative studies carried out thus far, their approach does face a number of important limitations [1]. For one, their democracy index needs to be unpacked and “its constituent components, such as freedom of the press, openness of the electoral process,” for example, tested against traditional determinants of democracy to determine whether one component provides more explanatory power than others. Another limitation of their data is the fact that it extends only to 2002. This should be updated today due to the rapid pace of ICT diffusion over the past several years. In addition, several scholars have criticized the Freedom House data with regards to conceptualization, measurement and aggregation issues (Munck and Verkuilen, 2002; Rydland et al., 2008). Furthermore, as discussed subsequently, there is little to no

variation in the Freedom House data, which makes meaningful statistical analysis more difficult. In contrast to the populist literature, the second school of thought disputes the majority of claims that exist vis-` -vis a the relationship between Internet and democracy. The counterarguments are based on both qualitative and quantitative research. In terms of qualitative research, several scholars argue that the Internet is merely an extension of the ruling class and centralized control [39], [40], [41], [34], [42]. According to Neuman, even if the increase of ICTs had led to an increase in the motivation to communicate - which he argues has not happened - then ICTs would have become centralized by government turning them into social control mechanisms [43]. Scholars who subscribe to this school of thought maintain that mass media information technologies discourage collective behavior, ”unless the rise in couch potatoes can be considered a social movement” [44], [45], [46]. In contrast to Snider’s argument about the mere potential of citizens being informed acting as a source of accountability, if power is measured by the potential for “monopoly and control over information and communication, it is evident that the new technology can become a dangerous facilitator of tyranny” [21], [2]. Indeed, while the Internet may enable citizens to subvert political hierarchy, Barber notes that with increased participation comes the peril of political and economic surveillance. The populist school of thought is often blind to “how easily liberating technologies become tools of repression” [2]. Bimber rejects the supposition that the Internet will have significant effects on public life, point out that “both theory and empirical evidence cast grave doubt on the communication-action connection at the core of the populist theory” [12]. Lippmann argued that the capacity of ICTs to recreate politics is constrained by human nature, ie. cognitive processing, and not by the technical properties of the media themselves [47]. The Internet, then, is no different than other ICTs even if the new medium differs from previous technologies in a fundamental way, namely allowing social bonding to occur asynchronously. In sum, the Internet is “hardly producing the first dramatic expansion in communication: telephone, radio, and television also expanded communication profoundly.” There seems no compelling reason to believe that the communication capacity of the Net will have such a dramatically different effect than have other advances in pointto-point and broadcast communication” [12]. Moreover, Page argues, new ICTs may very well overcome spatial distance but his far from sufficient for establishing vibrant forms of political communication and deliberation [48]. Furthermore, “if democracy is to be understood as deliberative and participatory activity on the part of responsible citizens, it will have to resist the innovative forms of demagoguery that accompany innovative technology and that are too often overlooked by enthusiasts [2]. Aristotle wrote that the basis of a democratic state is liberty. Barber adds that a “free society is free only to the degree that its citizens are informed and that communication among them is open and informed [2]. However, recent research and empirical work


confirms that governments increasingly have the upper hand in controlling and regulating the impact of the information revolution [49], [50], [51], [52], [53], [54], [55], [56], [57], [58]. As Goldsmith observes, “if governments can raise the cost of Net transactions, they can regulate the transactions” [59]. Beilock and Dimitrova found that countries with lower Freedom House scores for civil liberties had significantly lower Internet usage (even when controlling for economic development) [60]. De Mesquita and George Downs also argue that government elites (e.g., in Singapore) have learned to “stifle the bottom-up democratic potential of the Internet and still promote economic growth, contrary to Kedzie’s dictator’s dilemma argument” [61]. As Bimber notes, the “central theoretical problem for the populist claim is the absence of a clear link between increases in information and increases in popular political action” [12] To this end, McLuhan’s old dictum may be wrong: “the medium is not the whole message. Content matters, and there is simply no overwhelming reason to believe that a new medium will necessarily enhance the political quality of communicative content” [12]. In short, “technology need not inevitably corrupt democracy, but its potential for benign dominion cannot be ignored” [2]. In terms of quantitative analysis, Scheufele and Nisbet’s 2002 statistical study suggests that the Internet does not increase democracy. “Through linear regression, they find that mass media broadcasting (e.g., television, newspapers) plays a far more effective role than the Internet in promoting democratic citizenship” [1], [62]. Given that an established body of quantitative research on this topic has yet to materialize, Scheufele and Nisbet do caution against generalizing the results of their study, which focused exclusively on the United States. Other scholars interested in this line of research have questioned the supposed direction of causation drummed up by the populist school of thought. Using multiple measures of regime type, Milner’s statistical analysis demonstrates that, ceteris paribus, democracies permit much greater online access, both in terms of Internet users per capita and Internet hosts per capita [63]. To this end, the information revolution may merely be reinforcing pre-existing dynamics. Milner’s study uses data from 1991 − 2001 to measure the influence of regime type of adoption of the internet. This study attempts to address a slightly different question—whether there is a relationship between measures of democracy and ICT penetration—but we build on her work by extending the range of years to 2006 (although we begin measurement in 2000). We adopt a fixed effect model and control for the time component directly in the model. Beilock and Dimitrova develop a model to explain global country differences in Internet use using income, measures of freedom, region dummies, and development indices [60]. Their data is a cross section from 2001 and does not take democratic variables directly into account. Best and Wade ask the question closest to that addressed in this paper [1]. They ask whether Internet penetration has an effect on the level of democracy in a country. Their study

is global in scope and uses time series data from 1992 to 2002. We seek to build on their work by using data from 2000 to 2006 and using the World Bank Governance metrics as our measures of democratic tendency. Best and Wade combined the Freedom House metrics of political rights and civil liberties as their measure of democratic tendency. As described subsequently we feel the Freedom House data are not well suited to a regression study such as this one. A. Our Approach to Measuring Democratic Tendency Using the World Bank Governance Indicators Dahl characterizes a government with power vested in a plurality as follows [13]: 1) Control over governmental decisions about policy is constitutionally vested in elected officials. 2) Elected officials are chosen and peacefully removed in relatively frequent, fair and free elections in which coercion is quite limited. 3) Practically all adults have the right to vote in these elections. 4) Most adults also have the right to run for the public offices for which candidates run in these elections. 5) Citizens have an effectively enforced right to freedom of expression, particularly political expression, including criticism of the officials, the conduct of the government, the prevailing political, economic, and social system, and the dominant ideology. 6) They also have access to alternative sources of information that are not monopolized by the government or any other single group. 7) Finally, they have an effectively enforced right to form and join autonomous associations, including political associations, such as political parties and interest groups, that attempt to influence the government by competing in elections and by other peaceful means. The first four points largely describe procedural aspects of a democracy, whereas the last three points delineate the communication aspect necessary for a well-functioning democratic regime. In fact, Diamond goes further and notes that “[s]ome insist on a fairly robust (though still procedural) definition of democracy, like Dahl’s ‘polyarchy.’ By this conception, democracy requires not only free, fair, and competitive elections, but also the freedoms that make them truly meaningful (such as freedom of organization and freedom of expression), alternative sources of information, and institutions to ensure that government policies depend on the votes and preferences of citizens” [64]. Expanding on Dahl, Diamond has developed his own list of characteristics of a democracy [65]: 1) Control of the state and its key decisions and allocations lies, in fact as well as in constitutional theory, with elected officials (and not democratically unaccountable actors or foreign powers); in particular, the military is subordinate to the authority of elected civilian officials. 2) Executive power is constrained, constitutionally and in fact, by the autonomous power of other government in-







8) 9)


stitutions (such as an independent judiciary, parliament, and other mechanisms of horizontal accountability). Not only are electoral outcomes uncertain, with a significant opposition vote and the presumption of party alternation in government, but no group that adheres to constitutional principles is denied the right to form a party and contest elections (even if electoral thresholds and other rules exclude small parties from winning representation in parliament). Cultural, ethnic, religious, and other minority groups (as well as historically disadvantaged majorities) are not prohibited (legally or in practice) from expressing their interests in the political process or from speaking their language or practicing their culture. Beyond parties and elections, citizens have multiple, ongoing channels for expression and representation of their interests and values, including diverse, independent associations and movements, which they have the freedom to form and join. There are alternative sources of information (including independent media) to which citizens have (politically) unfettered access. Individuals also have substantial freedom of belief, opinion, discussion, speech, publication, assembly, demonstration, and petition. Citizens are politically equal under the law (even though they are invariably unequal in their political resources). Individual and group liberties are effectively protected by an independent, nondiscriminatory judiciary, whose decisions are enforced and respected by other centers of power. The rule of law protects citizens from unjustified detention, exile, terror, torture, and undue interference in their personal lives not only by the state but also by organized nonstate or anti-state forces.

Like Dahl, Diamond includes procedural aspects of a democratic regime (points one through three) and he enshrines what he considers essential communication requirements in points four through seven. These latter points can be characterized as approximating a “freedom of expression” or “political voice” aspect to democracy. We found the World Bank governance metric of “Political Voice and Accountability” to represent Diamond’s notion well in that it measures “perceptions of the extent to which a country’s citizen’s are able to participate in selecting their government, as well as freedom of expression, freedom of association, and a free media” [66]. Like all the World Bank Governance metrics, it was built from surveys and other sources of data within each country. These sources give an idea of “freedom of belief, opinion, discussion, speech, publication, assembly, demonstration, and petition” present in the country, although they do not measure the proliferation of channels of communication that Diamond enunciates. The World Bank governance indicators metrics are based on 35 data sources some of which yield “subjective or perceptionsbased data” including that from “household and firm survey

respondents, as well as thousands of experts working for the private sector, NGOs, and public sector agencies” [66]. In points eight through ten, Diamond gives a description of the role of law in a democracy. The World Bank has a governance metric that expresses some of this: “measuring perceptions of the extent to which agents have confidence in and abide by the rules of society, and in particular the quality of contract enforcement, property rights, the police, and the courts, as well as the likelihood of crime and violence” [66]. This approximates the World Banks “Rule of Law” metric and we propose it as an empirical measure of Diamond’s points eight through ten. The World Bank also has a metric measuring political stability: “perceptions of the likelihood that the government will be destabilized or overthrown by unconstitutional or violent means, including politically-motivated violence and terrorism” [66]. While not enumerated in either Diamond’s or Dahl’s lists, Amartya Sen postulated an empirical correlation between democratic regimes and political stability [67]. He notes both the “political incentives provided by democratic governance” to prevent crises and specifically that the “positive role of political and civil right applies to the prevention of economic and social disasters in general” [67]. Thus we investigate the World Banks “Political Stability” metric as another measure of democratic tendency. In measuring the relationship between ICT penetration and these democratic variables, it is clear that country wealth is a confounding factor that sound be taken into account: wealthier countries are both more likely to be democratic and to be the heavier users of both the Internet and the cell phone. We gathered gross domestic product (GDP) data from 2000 to 2006 from the International Monetary Fund. The GDP data is purchasing parity adjusted to be directly comparable between countries. Diamond notes that country size is highly correlated with regime type: “countries with populations under one million are much more likely to be both democracies and liberal democracies. Two-thirds of these countries are liberal democracies, while only 30 percent of countries with populations over one million are. Among the larger 150 countries, only half are democracies, while 70 percent of the small countries are. The countries with populations over one million are about twice as likely as small states to have an electoral authoritarian regime and half again as likely to have a closed authoritarian regime.” [64], [65]. Because of this, we included population in our models to control for country size. The population data for 2000 to 2006 was also gathered from the IMF.1 B. Limitations of The Data The ICT data is gathered from the International Telecommunications Union (ITU). The ITU requested the number of Internet and cell phone users from each country. This raises a host of questions about the reliability of the data since it
1 Both the IMF GDP and population data are available at http://www.imf. org/external/pubs/ft/weo/2008/01/weodata/


is self-reported by the country. Perhaps the country has an incentive to under or over-report? It is plausible that some countries have more reliable data collection mechanisms in place than others. With one report per country per year, the data are highly granular, but they do seem to follow steady trends, and steadily upwards in ICT adoption. The World Bank did not calculate governance metrics for 2001. We carried out simple linear interpolation to provide a data point for each country in this year. We made the decision to do this since there are only six time points from which to interpolate, and only one, the year 2000, falls before our missing data. This creates data that are “too smooth” for year 2001 since they have been created from our pre-existing data and this will make our results appear more precise than they in fact are. We argue that this effects is minor since the World Bank data are themselves aggregated from a large number of sources, and thus less subject to noise than using a single source would be. Other metrics of the level of democratic rights exist, such as the Freedom House “Freedom in the World” metric. Freedom House carries out an annual global survey of political rights and civil liberties. We choose not to use this as a measurement of democratic tendency for two reasons. We felt that the World Bank Governance Indicators could be well grounded in the theory of democracy as measures of democracy. Secondly, the Freedom House measures have some quantitative limitations. A certain amount of inertia is built into the measurements so that it is difficult for a country to move much from one year to the next. For both political rights and civil liberties a country is scaled from 0 to 7, giving only 8 possible outcomes for a country. Combining these two factors leads to a database that does not shift very much from year to year. In the years of our study, 2000 to 2006, of the 193 countries surveyed by Freedom House (after subtracting the 9 with missing values for both political rights and civil liberties for the entire time series), 105 had no change in their scores for political rights and 86 had no changes in their civil rights scores. The average variance of those that did exhibit some change from 2000 to 2006 was 0.48 for political rights and 0.32 for civil liberties. This means the majorities of countries, if they changed at all, changed by perhaps one point on the 8 point scale. Having more years of data, including 2007, would improve our modeling. At the time of this writing, the World Bank Governance metrics were not available for 2007. The IMF estimated some of the population number for some of the countries. It is likely this has the effect of providing population data that is smoother than it would otherwise be. Note also that both ICT measures, the Internet and Cell Phone use, are measures per 100 inhabitants. We emphasize that this must be carefully noted in interpretation of the regression results, since we use population as an independent variable. We also note that even though the ITU collects Internet use statistics for each country, what it really means to use the Internet can vary by country due to filtering, censoring, and other restrictions on access. The OpenNet Intiative at the Berkman Center for Internet and Society monitors the filtering

activity for 40 of the countries most actively engaged in repressing internet activity [68], [69]. Although not as extreme as the case of the Internet, cell phone use can be restricted by the government as well, and will differ from country to country. Zuckerman gives several examples of government crackdowns on mobile phone use: Belarus’s reported shutdown of their SMS network in March 2006, reports of Ethiopian cell phone blocking during the 2005 election protests, and Cambodian blocking of SMS for two days before their 2007 elections [7]. North Korea and Cuba were dropped from the study since official data is not reported for these countries. It is generally known that there is very little internet access in Cuba, and little to none in North Korea and these are both regimes with little democracy. If we had been able to include these countries in our estimation of the models, this would likely have bolstered our results.2 Details of the data cleaning and amalgamation process are on the study website at http://www.stodden.net/ ICTD. Our population data was obtained from the International Monetary Fund and contains a sparse number os missing values. The IMF has made estimates of their missing data to complete the dataset.3 III. E MPIRICAL M ODELING Our data comprises a panel containing N different times series each consisting of T observations. The number of countries, N , is 181, and T , the number of years in our study, is 7. A fixed effects model of our democratic measures’ effect on ICT penetration follows: ICTit = β0 + β1 RLit + β2 V Ait + β3 P Sit + β4 P OPit + β5 GDPit + β6 M Fit + γt T D + ξi CD +

i = 1, . . . , N, t = 1, . . . , T. In this paper we model the penetration of Internet or cell phone usage per 100 inhabitants, ICT , as a function of the World Bank democratic measures (Rule of Law, Voice and Accountability, and Political Stability)4 , country size, country wealth, and the male/female ratio in the country. ICT is one of “Internet Use” or “Cell Phone Use.”5 GDPit is the per capita Gross Domestic Product for country i at year t, adjusted for purchasing power parity.6 P OPit is the
2 Countries with missing values also tended to support our hypothesis: Afghanistan, Bhutan, Comoros, Kiribati, Serbia, St. Kitts and Nevis, TimorLeste, and Tonga. Cuba, Iraq, Montenegro, and North Korea simply did not furnish enough data for inclusion in the study. 3 A precise explanation of their data interpolation procedure was not readily available. See http://www.econstats.com/weo/V023.htm 4 Available at http://info.worldbank.org/governance/wgi/ 5 The data used in this study is available at http://www.itu.int/ITUD/ict/informationsharing/ 6 The data are available at http://www.imf.org/external/pubs/ft/weo/2008/01/weodata/index.aspx


population of country i at year t and M Fit is the male/female gender ratio.7 β0 is the intercept term for country i, and γt T D and ξi CD are a time effect and a country effect, respectively. The effect of time is controlled for by dummy variables: T Dt is 1 for year t and 0 otherwise. Similarly CDi is a dummy variable that is 1 for country i and 0 otherwise, controlling for the differences between countries. Finally, it is a disturbance term with 2 distribution N (0, σi ), which we assume to be uncorrelated across country cross sections. In this study there are N = 181 countries and T = 7 years. A. Autocorrelation in Panel Data A panel regression model of this type is subject to possible autocorrelation between subsequent observations because of the time series components. In a regression model as described above, it is possible to ’discover’ what Granger and Newbold [70] termed “spurious” relationships between the variables. That is, tests of significance on estimated coefficients may indicate a significant result, when in fact none is present. Granger and Newbold suggest economic time series data may be especially prone to autocorrelation since they tend to be non-stationary, in that it is not uncommon for the process generating the data to depend on the time it was sampled. For example, economic time series are commonly subject to seasonal or cyclical effects. Time series data that are not stationary will violate the assumptions of least squares regression since the variance of the error term will depend on time and thus introduce a bias into coefficient estimation.8 Granger and Newbold describe a high R2 value and a low Durbin-Watson statistic as warning signs that the estimated model may be yielding spurious results. Thus it is important to determine whether the time series data in this study are nonstationary. Our data are sampled yearly suggesting they may avoid annual cyclical effects. We analyze 181 countries from 2000 to 2006 (with 2001 interpolated for the World Bank variables) and thus have 7 values in each time series. As Granger and Newbold mention, finely sampled time series tend to exacerbate the cyclical effects and thus nonstationarity in the data. As is typical, the Augmented Dickey-Fuller test was used to assess nonstationarity in each of our time series. Since Internet and Cell phone use are increasing rapidly for the vast majority of the countries in our study, we measured the autoregressive structure of each time series as stationary around a trend line, and modeled with one lag. The null hypothesis is that the data are nonstationary. The test was applied to the ICT and World Bank data at the country level. For the Internet and mobile phone data the test rejected the null hypothesis of nonstationarity for 40 and 35 of 181 countries respectively, meaning that for around 20% of the Internet and cell phone penetration time series, nonstationarity is not evident.9 . When
gender ratio data was obtained from the Census Department’s International Database at http://www.census.gov/ipc/www/idb/tables.html 8 For a mathematically precise explanation see [70], p. 2. 9 The tests were performed at the 10% level
7 The

the same test was performed on the World Bank variables, 54, 62, and 60 of 181 rejected nonstationarity for Rule of Law, Voice and Accountability, and Political Stability, respectively (about one third of the data). Although there some evidence of stationarity, it appears that the majority of the time series included in this study are nonstationary, and it’s potential biasing of coefficient estimates is a concern.10 The typical remedy is the difference the data to remove the nonstationarity. Running the regressions in the above equation allowed us to carry out tests on the residuals directly to evaluate the level of autocorrelation. There are two regressions to be run in this study, modeling Internet penetration and cell phone usage. Typically the Durbin-Watson test with one lag is used to test for autocorrelation in the structure of the regression residuals and the regression with Internet use as a dependent variable was found to have autocorrelation present, and the cell phone penetration regression was not, with Durbin-Watson values of 1.86 and 2.01 respectively.11 The Durbin-Watson statistic is created by calculating d = PT 2 ( t=2Pt − t−1 ) , where t is the tth residual from the regrest 2 1 t sion. It follows that 0 < d < 4. A value of 2 indicates no autocorrelation. To test whether the Durbin-Watson test statistics could be considered equivalent to 2, the test in the R statistical software package was used [71]. The adjusted R2 values were 0.9246 and 0.9199, seeming to fit the Internet regression squarely into Granger and Newbold’s area of caution: a low Durbin-Watson statistic and a high R2 value, and also casts some suspicion on the cell phone regression. Granger and Newbold offer that until “a really satisfactory procedure is available, we recommend taking first differences of all variables that appear to be highly autocorrelated.” (p. 8.) We carried out this operation on both the Internet and cell phone use regressions because of the high R2 values and the nonstationary data in both regressions, even though the Durbin-Watson statistic did not suggest autocorrelation among the cell phone regression errors. The plots of the residuals for both regressions indicate possible heteroskedasticity. This suggests running the following differenced model: ∆ICTit = β0 + β1 ∆RLit + β2 ∆V Ait + β3 ∆P Sit + β4 ∆P OPit + β5 ∆GDPit + β6 ∆M Fit + γt T D + ξi CD +

i = 1, . . . , N, t = 2, . . . , T.

Running this model for differenced Internet penetration and differenced mobile phone use did not improve the Durbin10 Note that a combination of nonstationary time series may in fact be stationary. This is termed cointegration. 11 The p-values for the Durbin-Watson test are generated via a bootstrapping method and can fluctuate. In this case the p-values were 0.022 for the Internet regression and 0.962 for the cell phone regression.

TABLE I S UMMARY S TATISTICS FOR 2000 DATA Variable Internet Use per 100 Cell Use per 100 Rule of Law Voice & Accountability Political Stability GDP (PPP per capita) POP (millions) Gender Ratio (M/F) Mean 7.25 16.02 -0.07 -0.06 -0.08 8,998.10 33.20 100.4 Min 0 0 -2.02 -2.05 -2.73 229.36 0.04 85.3 Max 45.58 81.73 1.95 1.67 1.54 55,248.25 1267.43 212.3 TABLE II S UMMARY S TATISTICS FOR 2006 DATA Variable Internet Use per 100 Cell Use per 100 Rule of Law Voice & Accountability Political Stability GDP (PPP per capita) POP (millions) Gender Ratio (M/F) Mean 21.66 53.19 -0.07 -0.07 -0.08 12,147.70 35.67 100.4 Min 0.03 0.42 -2.00 -2.28 -2.31 195.43 0.051 84.3 Max 92.52 138.06 2.03 1.72 1.60 76,537.15 1314.48 218.5

Watson statistics. They became 2.16 and 1.92 respectively. The adjusted R2 values were reduced to 0.3686 and 0.4059. Both Durbin-Watson statistics reject the null hypothesis of no autocorrelation at the 5% level. A common method of controlling for autoregression is using a 2-stage least squares approach [74], [75]. In the first stage, the autoregressive structure in the residual is estimated using a model postulating that the autocorrelation has a single lag structure, specifically:


TABLE III R EGRESSION C OEFFICIENT E STIMATES FOR G LOBAL I NTERNET P ENETRATION Variable Rule of Law Voice & Accountability Political Stability GDP POP Gender Ratio Estimate 0.9018 0.9122 -0.3783 0.0004 -0.0488 0.8212 Standard Error 1.1193 0.8514 0.5345 0.0002 0.8123 0.9189 p-value 0.4207 0.2844 0.4793 0.0972* 0.9521 0.3718



A. Modeling Global ICT Penetration as a Function of Democratic Tendency We estimated two panel regressions of ICT penetration with controls for autocorrelation as discussed in the preceding section (differencing and the Durbin-Watson correction). Internet and cell phone use were modeled as functions of demographic variables along with control variables: ∆Internetit = β0 + β1 ∆RLit + β2 ∆V Ait + β3 ∆P Sit + β4 ∆P OPit + β5 ∆GDPit + β6 ∆M Fit +

where i is the ith residual from the initial regression. The first stage allows us to find an estimate of ρ, ρ using ˆ a least-squares model. In stage 2, ρ is used to remove the ˆ autocorrelation in the variables. Each variable, represented as V AR in the next equation, is then adjusted to create a new variable, adjV AR, using the formula: adjV ARi−1 = V ARi − ρV ARi−1 ˆ Since we are analyzing panel data with both time and country dimensions, implementation of the autocorrelation was applied at the country level using different estimates of ρ for the Internet and cell phone regressions. Carrying this out on the differenced data increased the Durbin-Watson statistics to 2.24 and 2.18, and autocorrelation is still detected by this test at the 5% level. The adjusted R2 statistics were 0.4292 and 0.3599. This was our final model analyzed in the following section. Since autocorrelation has not been eradicated from the data we interpret our results cautiously and look for corroboration. IV. R ESULTS AND F INDINGS We used the R statistical package to estimate these models (version 2.7.2) [71]. The complete set of code and data used in this study can be found at http://www.stodden.net/ICTD. Tables I and II present summary statistics of the dependent variables (Internet and cell phone use per 100 country inhabitants) and the World Bank metrics for the countries in 2000 and 2006 respectively.
12 Testing the residuals for autocorrelation using the Durbin-Watson test is the typical procedure, when the sample is large. It is also possible to test for cointegration: whether the combination of time series is stationary. See [72] and [73].

γt T D + ξi CD +


i = 1, . . . , N, t = 2, . . . , T. The regression coefficients for Internet penetration are given in Table III. The coefficients on the individual country and time dummy variables are not included for space reasons.13 The most significant coefficient was GDP, and none of the democratic measures were significant. The positive coefficient on GDP confirms our intuition that wealthier countries have higher levels of Internet use. The same regression was run for cell phone penetration, and the coefficient estimates are presented in Table IV. Cell phone use appears not to associated with wealth as Internet use is, and the male to female gender ratio in the country is strongly associated with increases in cell phone use, as is the level of political stability. The influence of the gender ratio may be driven by outlier countries: Most countries had a male/female gender ratio of a little less than 100, implying slightly more females than males in the population. The coefficient of 4.88 implies that
13 The complete regression results can be found online at http://www. stodden.net/ICTD.

TABLE IV R EGRESSION C OEFFICIENT E STIMATES FOR G LOBAL C ELL P HONE P ENETRATION Variable Rule of Law Voice & Accountability Political Stability GDP POP Gender Ratio Estimate -1.4120 -2.458 2.2823 -0.0002 0.1225 4.8839 Standard Error 2.0333 1.5946 1.0044 0.0005 1.4327 1.7342 p-value 0.4876 0.1236 0.0233** 0.7061 0.9319 0.0050*** TABLE VI C ELL P HONE U SE PER 100 I NHABITANTS FOR H IGHER G ENDER R ATIO C OUNTRIES , 2000 AND 2006 Rank 1 2 3 4 5 6 7 8 9 10 Country United Arab Emirates Qatar Kuwait Oman Bahrain Saudi Arabia Maldives Bhutan Jordan Djibouti 2000 43.98 19.90 21.74 6.63 30.61 6.40 2.83 0 7.72 0.04 Country United Arab Emirates Qatar Kuwait Maldives Bahrain Oman Saudi Arabia Bhutan Jordan Grenada 2006 118.51 109.6 91.49 87.88 122.88 69.59 78.05 9.77 74.4 44.59

TABLE V T OP 10 M ALE /F EMALE R ATIOS , 2000 AND 2006 Rank 1 2 3 4 5 6 7 8 9 10 Country United Arab Emirates Qatar Kuwait Oman Bahrain Saudi Arabia Maldives Bhutan Jordan Djibouti 2000 212.3 198.1 150.3 131.3 129.9 125 117.3 112.3 109.9 107.1 Country United Arab Emirates Qatar Kuwait Maldives Bahrain Oman Saudi Arabia Bhutan Jordan Grenada 2006 218.5 202.5 152.3 127.3 126.7 124.7 120.5 111.0 110.2 108.1

as the gender ratio increases by about 5, cell phone usage per 100 inhabitants will increase by one phone. Throughout the years studied, roughly 7 of the 10 countries with the highest male/female ratio each year were located in the middle east, and the ratios at that end of the distribution dwarfed the other countries’. As displayed in Table VI, it is plausible some of these values are extreme enough to have a large impact on the regression fit, although why this did not occur in the internet regression is not clear. It is also possible the very high gender ratio values represent a ’middle east effect’ since a number of those countries are highly represented in the top 10 gender ratio values. This implies that the coefficient on the gender ratio variable could represent a high growth in cell phone use in the middle east. Table VII gives the 2000 and 2006 cell phone data for these countries. The coefficient indicates that as the rate of change in the proportion of men increases, so does the rate of change in cell phone use. Interestingly, the greater political stability and the lower the perceived threat of violence, the greater cell phone penetration. This may represent infrastructural stability if associated with political stability and thus a measure of investor’s confidence. It is not clear why this factor would not therefore also be associated with an increase in Internet use. Perhaps cell phones are easier to proliferate than access to the Internet and so a smaller increase in political stability encourages cell phone increase before Internet increase. As shown in Tables I and II, the average cell phone penetration in 2000 was about 16 phones per 100 inhabitants and in 2006 it was about 53, nearly a three-fold increase. This is a high rate of increase but, notably, the countries listed in Table VI (those with the highest male to female gender ratios) had much higher than average growth in cell phone penetration. This is quantified in the significant coefficient in the regression in Table IV, while allowing for the included

confounding factors. The World Bank measure for political stability is also significantly positively correlated with increased cell phone use. This finding suggests that political instability is related to the mass diffusion of cell phone usage. In other words, an increase in cell phone availability could increase the perceived likelihood that the government will be destabilized or overthrown by unconstitutional or violent means. In their statistical analysis, Mansfield and Snyder find that the process of democratization itself is indeed a destabilizing one [76]. “Certainly, the virtues of working democratic structures do not translate into a carefree path to the stabilization of democracy” [77]. This finding also supports the arguments presented by Rosenau and Johnson, as well as Shirky, who opine that the Internet can be used as a tool by civil society to organize collectively to effect political change [25], [78]. B. Modeling ICT Penetration as a Function of Democratic Tendency Among the Most and Least Affluent Countries Examining ICT penetration for different strata of wealth may help isolate effects that are characteristic of those groups. Since wealth is a driver of investment one would expect GDP to play a role in the country’s readiness and ability to adopt new communication technologies [79], [80], [81]. As established in the literature we also found increases in GDP to be associated with increases in Internet use. We choose to examine ICT penetration in both the top and bottom 20% of countries by 2006 GDP more closely. Our focus on these groups, in particular the bottom quintile, is driven by Mansfield and Snyder’s work theorizing the instability of emergent and transitional regimes and the existence of the global digital divide [82]. Table VII lists the countries that fall into each of these groups. As in the previous section we fit a model with an ICT penetration measure as the explanatory variable, and measures of democratic tendency and controls as independent variables for a panel regression over years 2000 to 2006. For the top wealthiest quintile of countries our model did not yield statistically significant results for Internet penetration. Table VIII gives the coefficient estimates. Since these countries are exceptionally wealthy and relatively stable politically it may not be a surprise that GDP is not a driver of Internet use, and

TABLE VII T OP AND B OTTOM 20% OF C OUNTRIES BY GDP IN 2006 Top 20% Countries Qatar Luxembourg Brunei Darussalam Norway Singapore United States Ireland Switzerland Hong Kong, China Kuwait Iceland Canada Netherlands Austria Denmark United Arab Emirates Sweden Australia Belgium United Kingdom Finland Germany Japan France Bahrain Italy Spain Taiwan, China Greece Cyprus New Zealand Slovenia Israel Bahamas Korea (Rep.) Saudi Arabia Czech Republic Bottom 20% Countries Zimbabwe Congo (Dem. Rep.) Liberia Burundi Guinea-Bissau Afghanistan Sierra Leone Niger Central African Rep. Ethiopia Malawi Eritrea Mozambique Togo Rwanda Uganda Myanmar Mali Madagascar Guinea Comoros Tanzania Nepal Burkina Faso Lesotho Bangladesh Gambia Haiti Zambia Ghana Sao Tom´ & Principe e Benin Kenya Senegal Cambodia Chad Cˆ te d’Ivoire o TABLE IX I NTERNET P ENETRATION , B OTTOM 20% OF GDP (2006) Variable Rule of Law Voice & Accountability Political Stability GDP POP Gender Ratio Coefficient 0.3702 -0.8115 -0.2583 -0.0002 -0.0309 0.4026 Standard Error 0.4081 0.3656 0.1822 0.0014 0.2784 0.3235 p-value 0.3660 0.0281** 0.1585 0.9053 0.9117 0.2155

TABLE X C ELL P HONE P ENETRATION , T OP 20% Variable Rule of Law Voice & Accountability Political Stability GDP POP Gender Ratio Coefficient 2.354 4.012 5.071 -0.0008 -2.896 2.8300


GDP (2006) p-value 0.7029 0.3984 0.0612* 0.1071 0.6558 0.5182

Standard Error 6.161 4.736 1.887 0.0005 6.483 4.3690

TABLE VIII I NTERNET P ENETRATION , T OP 20% OF GDP (2006) Variable Rule of Law Voice & Accountability Political Stability GDP POP Gender Ratio Coefficient -0.3709 4.771 -1.0344 0.0003 1.6791 -1.7979 Standard Error 5.4055 4.0981 2.2760 0.0004 5.5724 3.7568 p-value 0.945 0.246 0.650 0.450 0.764 0.633

nor are the measures of democratic tendency, even though our subsets contains countries with varying levels of autocratic control. Among the least wealthy quintile we fit the same panel regression model as above. Table IX gives the regression coefficient estimates. Our predictors did not yield highly significant coefficient estimates with the exception of the World Bank voice and Accountability metric. Voice and Accountability is negatively correlated with Internet penetration: implying that when countries notch up in the Voice and Accountability ranking, the use of the Internet increases. This seemingly paradoxical finding may be explained when note that our analysis is restricted the the lowest quintile of country in wealth. These countries experience disproportionately greater

political turmoil and it may be the case that countries with higher Voice and Accountability rankings have been reluctant to permit the growth of the Internet in their milieu. Table X gives the coefficient estimates from the panel regression for cell phone penetration for the top 20% of wealthiest countries. There is a statistically significant effect in the World Bank metric of Political Stability: greater Political Stability is associated with an increase in cell phone use per inhabitant. This result seems intuitive as political stability is historically associated with greater investment in communications infrastructure and is consistent with our earlier regression on cell phone use. This suggests the wealthiest countries may be driving the correlation between political stability and cell phone penetration. None of the other variables were found to have a statistically significant relationship with cell phone penetration in the wealthiest countries. Among the poorest countries, growth in Voice and Accountability had a statistically significant negative effect on growth in cell phone penetration. Although consistent with the Internet penetration regression results for this group of countries, the paradox remains as to the increase in per capita cell phone use as Voice and Accountability decreases. It is plausible citizens desire newer forms of ICT when Voice and Accountability is restricted. Another explanation may be that although not all countries in the bottom quintile are autocratic, a significant proportion are and cell phone use may facilitate the mobilization, organization and coordination of resistance against autocratic rule. Interestingly, the voice and accountability metric is not a significant predictor globally, yet is significant for types of ICT among the poorest countries. V. C ONCLUSIONS AND F UTURE R ESEARCH This paper is the first to our knowledge that uses recent Internet and cell phone use data in an empirical study of their relationship to democratic tendency. Previous studies uses measures of Internet use that ended in 1993 [26], [38]. Best and Wade’s data reached only to 2002. This paper is also the first to the best of our knowledge that measures ICT diffusion

TABLE XI C ELL P HONE P ENETRATION , B OTTOM 20% OF GDP (2006) Variable Rule of Law Voice & Accountability Political Stability GDP POP Gender Ratio Coefficient 0.3702 -0.8115 -0.2583 -0.0002 -0.0309 0.4026 Standard Error 0.4081 0.3656 0.1822 0.0014 0.2784 0.3235 p-value 0.3660 0.0281** 0.1585 0.9053 0.9117 0.2155

as a function of democratic indicators. Previous research has focused on whether ICT use predicts democratic measures. These results support Bimbers assertions that the structure of political power has not been revolutionized or transformed into a new epoch of democracy [12]. Evidence can be found for both the populist thread in the literature and the notion that ICTs may act as an extension of the ruling class. We found a statistically significant positive relationship between the rate of diffusion of the cell phone and the World Bank’s “Political Stability” measure capturing perceptions regarding the likelihood that a government will be destabilized or overthrown by unconstitutional or violent means. Political Stability continues to have a significant positive relationship with the rate of cell phone use among the most affluent 20% of countries, but that relationship does not hold among the wealthiest countries. The “Voice and Accountability” indicator which measures perceptions of the extent to which citizens are able to participate in selecting their government, as well as freedom of expression, association and of the media, was a significant negative predictor of the rate of cell and Internet diffusion among the poorest quintile of countries. The diffusion of ICT access did not have any significant influence on “Rule of Law” while the increasing availability of cell phones were shown to have no influence on any of the three World Bank indicators. Recall that the Rule of Law metric measures perceptions of the extent to which individuals have confidence in and abide by the rules of society—in particular the quality of contract enforcement, property rights, the police, and the courts as well as the likelihood of crime and violence. This measure was originally included in the analysis based on Diamond’s research on the characteristics of democracy. However, the findings here suggest that the increase in “Rule of Law” perceptions does not influence ICT penetration, either positively or negatively. In other words, perceptions regarding the “Rule of Law” may be framed and influenced by factors other than widespread ICT use. The populist school of thought believes ICT diffusion will decentralize access to communication and information while increasing citizen access [12], while Hill and Hughes claim that perhaps the Internet will bring about “a wider democratic revolution in the world” [22]. In our modeling Internet diffusion was not predicted by our measures of democracy, implying this revolution has not yet arrived. The rate of cell phone diffusion was predicted by higher rates of the “Political Stability” metric. Recall that cell phone are much more widely

used globally than the Internet, so this may be a function of cell technology’s earlier foothold than the Internet’s. As Internet diffusion catches up to that of cell phone, the democratic metrics may be found to be predictors of this diffusion. This is not necessarily inconsistent with the populist thread in the literature as, according to Bimber, other scholars believe that the Internet may have a “transformative potential” because of the “one-to-one interaction among citizens and between citizens and government” it creates [12]. A paradox is created for the populist school in the finding that “Voice and Accountability” is negatively correlated with cell phone diffusion. This can be interpreted in favor of the argument that ICT diffusion can be centralized by government turning them into social control mechanisms [43]. As the rate of Voice and Accountability increases, the rate of diffusion of ICTs decreases among the poorest and least developed countries. Rates of cell phone use increase globally with decreases in political stability, lending further support to the thesis of ICTs as an extension of ruling class control. This paradox may be resolved is we consider Page’s view in 1995, that ICT diffusion may be still too nascent and insufficient to generate a well functioning system of political communication and deliberation [48]. It appears we are still too early to expect a close relationship between a vibrant public sphere and ICTs globally. As Bimber states, the “central theoretical problem for the populist claim is the absence of a clear link between increases in information and increases in popular political action” [12]. This paper provides evidence of the existence of this problem and the need to develop our understanding of this dynamic further. It would be interesting to tie this research more closely to development, in line with Sen’s reasoning that “Developing and strengthening a democratic system is an essential component of the process of development,” could extend the empirical analysis in a fruitful direction. This might mean specifically testing whether the order in which political and civil rights are extended as a country emerges from an autocracy affects the rate of development. This could provide a setting in which to test the “Lee Thesis,” that political rights should be withheld until economic development is achieved. It may be valuable to explore empirical issues further. Modeling the autocorrelation structure with more than one lag may help reduce autocorrelation. It would also be interesting to test for cointegration among these variables. Certainly documented feedback loops exist between our independent variables, such as GDP and measures of democracy, and taking this explicitly into account may improve the modeling [83], [26]. It is also plausible that feedback loops exist between ICTs and democratic measures and future modeling could accommodate this. Further research into the modeling aspects could estimate models including variables that control for the different manifestations of cell phone and Internet use in different countries. A more comprehensive model might explore possible non-linearities: whether countries with low ICT adoption rates have different patterns of democratic tendency than those with high adoption rates.


ACKNOWLEDGMENT The authors would like to thank Michael Best and Colin Maclay for helpful discussion and the anonymous reviewers for constructive criticism. Sadia Ahsanuddin and Kristen Lovin provided outstanding research assistance. R EFERENCES
[1] M. Best and K. Wade, “The internet and democracy: Global catalyst or democratic dud?” June 2006, sam Nunn School of International Affairs. [2] B. Barber, “Three scenarios for the future of technology and strong democracy,” Political Science Quarterly, vol. 113, no. 41573-589, 1999. [3] P. Schmitter and T. Karl, “What democracy is . . . and is not,” Journal of Democracy, vol. 2, no. 3, 1991. [4] P. Norris, “Political protest in fragile states,” July 2006, international Political Science Association World Congress. [5] F. Zakaria, “The rise of illiberal democracies,” Foreign Affairs, November 1997. [6] Huber, D. Rueschemeyer, and J. Stephens, “The impact of economic development on democracy,” Journal of Economic Perspectives, vol. 7, no. 3, pp. 71–85, 1993. [7] E. Zuckerman, “Draft paper on mobile phones and activism,” 2007. [Online]. Available: http://www.ethanzuckerman.com/blog/2007/04/09/ draft-paper-on-mobile-phones-and-activism/ [8] D. Kidd, Indymedia.org: a new communications commons. Routledge, 2003, pp. 47–69. [9] A. Scott and J. Street, “From media politics to e-protest,” Information, Communication and Society, vol. 3, no. 2, pp. 215–40, 2000. [10] S. Andrew, “The control revolution: How the internet is putting individuals in charge and changing the world,” The Century Foundation, Tech. Rep., 2000. [11] R. Clarke, “Information technology: Weapon of authoritarianism or tool of democracy?” August 1994, iFIP World Congress. [12] B. Bimber, “The internet and political mobilization - research note on the 1996 election season,” Social Science Computer Review, vol. 16, no. 4, pp. 391–401, 1998. [13] R. Dahl, Democracy and its Critics. New Haven, CT: Yale University, 1989. [14] H. Rheingold, Virtual Reality. New York, NY: Summit Books, 1991. [15] R. Anderson, T. Bikson, S. Law, and B. Mitchell, “Universal access to e-mail: Feasibility and societal implications,” RAND, Tech. Rep., 1995. [16] A. Etzioni, The Spirit of Community: Rights, Responsibilities, and the Communitarian Agenda. New York, NY: Crown Publishers, 1993. [17] J. Pavlik, New Media Technology: Cultural and Commercial Perspective. Boston, MA: Allyn and Bacon, 1996. [18] L. Grossman, The Electronic Republic. New York: Viking, 1995. [19] G. Browning, Electronic Democracy: Using the Internet to Influence Politics. Wilton, CT: Online Inc., 1996. [20] N. Negroponte, Being Digital. New York, NY: Vintage, 1995. [21] J. Snider, “New media: Potential information and democratic accountability: A case study of governmental access community media,” August 1996, american Political Science Association Convention. [22] K. Hill and J. Hughes, “Computer mediated political communication: The usenet and political communities,” Political Communication, vol. 14, pp. 3–27, 1999. [23] A. Corrado and C. Firestone, Elections in Cyberspace: Toward a New Era in American Politics, A. Corrado and C. Firestone, Eds. Washington, DC: Aspen Institute, 1996. [24] C. Steele and A. Stein, Communications revolutions and international politics. Albany, NY: State University of New York Press, 2002. [25] J. Rosenau and D. Johnson, Information technology and turbulence in world politics. State University of New York Press, 2002. [26] C. Kedzie, “Communication and democracy: Coincident revolutions and the emergent dictator’s dilemma,” RAND, Tech. Rep., 1997. [Online]. Available: http://www.rand.org/publications/RGSD/RGSD127 [27] S. Lipset, “Some social requisites of democracy: Economic development and political legitimacy,” American Political Science Review, vol. 53, pp. 69–105, 1959. [28] R. Jackman, “On the relation of economic development to democratic performance,” American Journal of Political Science, vol. 17, pp. 611– 21, 1973. [29] K. Bollen, “Political democracy and the timing of development,” American Social Review, vol. 44, pp. 572–87, 1979.

[30] R. Burkhard and M. Lewis-Beck, “Comparative democracy: the economic development thesis,” American Political Science Review, vol. 88, pp. 903–10, 1994. [31] J. Longdregan and K. Poole, “Poverty, the coup trap, and the seizure of executive power,” World Politics, vol. 42, pp. 151–83, 1990. [32] ——, “Does high income promote democracy?” World Politics, vol. 49, pp. 1–30, 1996. [33] B. Geddes, “What do we know about democratization after twenty years?” Annual Review of Political Science, vol. 2, pp. 115–44, 1999. [34] F. Webster, Globalization, information and change. State University of New York Press, 2002. [35] R. Reich, The Work of Nations: Preparing ourselves for 21st century capitalism. Vintage, 1992. [36] T. Friedman, The Lexus and the Olive Tree: Understanding Globalization. Anchor, 1999. [37] L. Waverman, M. Meschi, and M. Fuss, “The impact of telecoms on economic growth in developing countries,” Centre for Economic Policy Research, Tech. Rep., March 2005. [38] D. Richards, Making the national international: Information technology and government respect for human rights. Albany, NY: State University of New York Press, 2002. [39] D. Barney, Prometheus Wired: The Hope for Democracy in the Age of Network Technology. University of Chicago Press, 2000. [40] A. Mazrui and R. Ostergard, Technology transfer in the computer age: The African experience. State University of New York Press, 2002. [41] D. Strienstra, Gender, women’s organizing, and the Internet. Albany, NY: State University of New York Press, 2002. [42] S. Kalathil and T. Boas, Open Networks, Closed Regimes. Carnegie Endowment for International Peace., 2003. [43] W. Neuman, The future of the mass audience. Cambridge University Press, 1991. [44] T. Eyck, “Does information matter? a research note on information technologies and political protest,” Social Science Journal, vol. 38, pp. 147–160, 2001. [45] N. Postman, Amusing ourselves to death. New York, NY: Vintage Books, 1985. [46] H. Schiller, Culture Inc. Oxford University Press, 1989. [47] W. Lippmann, Public Opinion. New York, NY: Macmillan, 1934. [48] Page, Who Deliberates? Mass Media in Modern Democracy. Chicago, IL: University of Chicago Press, 1996. [49] G. Cherian, “The limits of singapore’s ‘light touch’ web regulation,” 2008, politics 2.0 International Conference. [50] R. Deibert, J. Palfrey, R. Rohozinski, and J. Zittrain, Access Denied: The Practice and Policy of Global Internet Filtering. The MIT Press, 2008. [51] S. Oats, “Internet and democracy in russia,” April 2008, politics 2.0 International Conference. [52] R. Singel, “Seeking tighter censorship, repressive states target web 2.0 apps,” Wired Magazine, March 2008. [Online]. Available: http://blog.wired.com/business/2008/03/etech-what-happ.html [53] J. Zittrain, The Future of the Internet—And How to Stop It. Yale University Press, 2008. [54] S. Mydans, “Myanmar junta unplugs internet,” The International Herald Tribune, October 2003. [Online]. Available: http://www.iht.com/ articles/2007/10/04/asia/04info.php [55] J. Goldsmith and T. Wu, Who Controls the Internet? Illusions of a Borderless World. Oxford University Press, 2006. [56] L. Lessig, Code: Version 2.0. Basic Books, 2006. [57] D. Drezner, “The internet’s effect on state-society relations,” March 2006, international Studies Association Convention. [58] T. Price, Cyber Activism: Advocacy Groups and the Internet. Washington, DC: Foundation for Public Affairs, 2000. [59] J. Goldsmith, “Regulation of the internet: Three persistent fallacies,” Chicago-Kent Law Review, vol. 73, December 1998. [60] R. Beilock and D. Dimitrova, “An exploratory model of inter-country internet diffusion,” Telecommunications Policy, vol. 27, pp. 237–252, 2003. [61] B. Mesquita and G. Downs, “Development and democracy,” Foreign Affairs, September 2005. [62] D. Scheufele and M. Nisbet, “Being a citizen online: New opportunities and dead ends,” The Harvard International Journal of Press/Politics, vol. 7, no. 3, pp. 55–75, 2002. [63] H. Milner, “The digital divide: The role of political institutions in technology diffusion,” August 2003, american Political Science Association.

[64] L. Diamond, “elections without democracy: Thinking about hybrid regimes,” Journal of Democracy, vol. 13, no. 2, April 2002. [65] ——, Developing Democracy: Toward Consolidation. John Hopkins University Press, 1999. [66] D. Kaufmann, A. Kraay, and M. Mastruzzi, “Governance matters vii: Aggregate and individual governance indicators 1996-2007,” World Bank, Tech. Rep., 2007. [67] A. Sen, Development as Freedom. Oxford University Press, 2001. [68] “Opennet initiative.” [Online]. Available: http://opennet.net/ [69] R. Faris and N. Villeneuve, Access Denied: The Practice and Policy of Global Internet Filtering. MIT Press, 2008, ch. 1. [70] C. Granger and P. Newbold, “Spurious regressions in econometrics,” Journal of Econometrics, vol. 2, pp. 111–120, 1974. [71] “R.” [Online]. Available: http://www.r-project.org/ [72] C. Granger, “Some properties of time series data and their use in econometric model specification,” Journal of Econometrics, vol. 16, pp. 121–130, 1981. [73] ——, “Co-integration and error correction: Representation, estimation and testing,” Econometrica, vol. 2, pp. 251–176, 1987. [74] T. Amemiya, “The nonlinear two-stage least squares estimator,” Journal of Econometrics, vol. 2, pp. 105–110, 1974. [75] R. Cumby, J. Huizinga, and M. Obstfeld, “Two-step two-stage least squares estimation in models with rational expectations,” Journal of Econometrics, vol. 21, pp. 333–355, 1983. [76] E. Mansfield and J. Snyder, “Conceptualizing and measuring democracy:evaluating alternative indices,” International Organization, vol. 56, no. 2, pp. 297–337, 2001. [77] H. Barnes, “The contribution of democracy to rebuilding postconflict societies,” The American Journal of International Law, vol. 95, no. 1, pp. 86–101, 2001. [78] C. Shirky, Here Comes Everybody: The Power of Organizing without Organizations. Penguin Press, 2008. [79] C. Argevou, “Ols bias in a nonstationary autoregression,” Information Technology for Development, vol. 8, no. 1, pp. 15–28, 1998. [80] B. Oyelaran-Oyeyinka and K. Lai, “Internet diffusion in sub-saharn africa: A cross-country analysis,” Telecommunication Policy, vol. 8, no. 1, pp. 507–527, 1998. [81] J. Gibbs, K. Kraemer, and J. Dedrick, “Environment and policy factors shaping global e-commerce diffusion: A cross country comparison,” The Information Society, vol. 19, no. 1, pp. 5–18, 1993. [82] N. Kshetri and N. Dholakia, ENCYCLOPEDIA OF INFORMATION SCIENCE AND TECHNOLOGY, 2nd edition. Idea Group Publishing, 2008. [83] P. Norris, Digital Divide: Civic Engagement, Information Poverty, and the Internet Worldwide. Cambridge University Press, 2001.


A review of the research on mobile use by micro and small enterprises (MSEs)
Jonathan Donner and Marcela Escobari
Like landlines, mobile phones allow people to communicate at a distance and exchange information instantaneously. Thus, there is significant potential for mobile use to increase MSE productivity. However, since the dynamics underpinning this potential are nuanced, and since current supporting evidence is scarce and methodologically heterogeneous [9], it is important to rigorously examine mobile use by MSEs. For example, there is a difference between using a mobile to serve existing customers more effectively, and using it to start a new business. There is a difference between using a mobile to check market prices and using it bypass a middleman who carries goods to market. Popular press and practitioner reports generally fail to make these distinctions. Fortunately, a small but methodologically diverse set of research studies have examined mobile use by MSEs in detail. This paper offers a systematic review of this existing literature, identifying known patterns of mobile use, as well as some important gaps in the research. The review employs distinct foci. First, it offers an assessment of how mobile use influences the internal process of an enterprise, using Porter’s value chain model [23]. Second, it offers a corresponding assessment of how mobile use influences the network of relationships external to the enterprise—the value system [23] of producers, traders, wholesalers, retailers and end-customers. Finally, it explores two elements unique to mobile communication—the increased spatial and temporal mobility afforded by wireless devices, and the resulting blurring of the personal and the professional spheres—to assess how MSE mobile use differs from landline use. II. MICRO AND SMALL ENTERPRISES (MSES) Working definitions of MSEs vary from country to country and from researcher to researcher [24]. This analysis defines an MSE as any non-farm i enterprise, formal or informal, with less than 50 employees, including sole proprietorships, parttime businesses, and home-based businesses. The size thresholds draw on Mead and Leidholm, [16], who note that the absolute majority of such enterprises in the developing world are sole proprietorships, and that firms with less than 10 employees substantially outnumber larger enterprises. A number of factors distinguish the term MSE (micro and small enterprise) from SME (small and medium enterprise). The terms MSE and SME are acronyms, each combining two distinct sizes of enterprises into a single reference. However,

Abstract— The paper offers a systematic review of 14 studies of the use of mobile telephony by micro and small enterprises (MSEs) in the developing world, detailing findings about changes to enterprises’ internal processes and external relationships, and findings about mobile use vs. traditional landline use. Results suggest that there is currently more evidence for the benefits of mobile use accruing mostly (but not exclusively) to existing MSEs rather than new MSEs, in ways that amplify existing material and informational flows rather than transform them. The review presents a more complete picture of mobile use by MSEs than was previously available to ICTD researchers, and indentifies priorities for future research, including comparisons of the impact of mobile use across subsectors of MSEs and assessments of use of advanced services such as mobile banking and mobile commerce.

Index Terms—Business Economics, Communication, ICTD, Mobile Communication, MSE, Social Factors

his paper presents a systematic review of fourteen studies of the use of mobile telephony by Micro and Small Enterprises (MSEs) in the developing world [1-14]. The majority of non-agricultural enterprises in the developing world have ten or fewer employees [15, 16]. These MSEs employ up to 25% of working-age adults in some countries [16], and while the contribution of MSEs to aggregate economic growth remains a matter of debate [17], their importance to household livelihoods and poverty alleviation is undeniable. Thus, MSEs are the focus of programs at many of the world’s largest development institutions [18]. Since the year 2000, the spread of mobile telephony across the developing world has raised hopes among policymakers that MSEs will benefit from easier access to telecommunications. The successful entrepreneur, suddenly enabled by his mobile phone, has been given a prominent role in the global development narrative and become a semiregular fixture in both the popular press [19, 20] and practitioner media [21, 22].
Manuscript version received February 20, 2009. J. Donner is with Microsoft Research India, jdonner@microsoft.com M Escobari is with the Center for International Development, marcela_escobari@harvard.edu




with no commonly accepted definitions of the thresholds between micro, small, and medium, there are often implicit conceptual overlaps between the acronyms. Unlike SMEs, the majority of MSEs are informal enterprises. Once again, however, there is no universal standard to determine what makes an enterprise informal vs. formal [6]. In many cases, even the term entrepreneur may be a bit of a romantic misnomer. Evidence suggests that among MSEs, only a small minority of enterprises are poised for growth; most remain small or struggle to survive, and yield a low return on labor and capital [17, 25]. Though significantly less growth-oriented and productive (on average) than SMEs and other larger firms [17], MSEs share a basic similarity with all enterprises; each combines investments in capital with some labor (their own, their families’ or their employees) in the hopes of yielding a product or service whose market value exceeds the cost of those inputs. Thus, there has always been a thread in the ICTD literature that seeks to understand how various technologies could be used advantageously by MSEs [26-28]. Prior to the widespread introduction of the mobile into the developing world, the landline’s importance in this regard was already clear: Phones are the information-related technology that has done the most to reduce costs, increase income and reduce uncertainty and risk. Phones support the current reality of informal information systems, they can help extend social and business networks, and they clearly substitute for journeys and, in some cases, for brokers, traders and other business intermediaries. They therefore work “with the grain” of informality yet at the same time help to eat into the problems of insularity that can run alongside. Phones also meet the priority information needs of this group of communication rather than processing of information [27: 18] The quotation focuses directly on the basic tasks of running a business—reducing costs, increasing income, managing risk—and links them to core functions of mediated communication technologies, particularly the substitution for journeys. As demonstrated elsewhere, [26] the key is increased productivity. III.

Particularly focused and powerful evidence appears in Jensen’s [10] research on the fishermen of Kerala. Working with five-year time series data at three fish markets in coastal India, Jensen and his team found that “the adoption of mobile phones by fishermen and wholesalers was associated with a dramatic reduction in price dispersion, the complete elimination of waste, and near-perfect adherence to the Law of One Price. Both consumer and producer welfare increased.” [10: 879] Soon after the introduction of mobile coverage, fishermen bought mobiles and accumulated lists of up to 100 buyers in their handsets’ address books; subsequently, while still at sea, fishermen could call a range of possible landing points and buyers in order to determine the best price and best place to sell their catch. By contrast, Jagun, Heeks, and Whalley’s [9] examination of the mobile’s role in mediating supply chains in the Nigerian market for traditional hand-woven ceremonial cloth is broad in scope. It offers a multidisciplinary literature review, a conceptual framework articulating effects at multiple levels, and a detailed case study. They describe “process” benefits to mobile use, as calls at a distance reduce the time of trades and replace costly journeys. They also describe “structural” impacts; finding no disintermediation of traders, but rather an intensification of their role. Traders are more likely to have mobiles than the less prosperous weavers in the supply chain, and thus are better positioned to coordinate with a wider range of downstream customers and to maintain a more dynamic and responsive set of relationships with weavers. For example, weavers previously had to pay cash to get their supplies. Mobiles give weavers access to credit by enabling calls on their behalf to fabric vendors by traders, who vouch for the veracity of weavers’ orders, and promise to cover the costs of the fabric in advance of the completion of the weavers’ work. IV. METHODS AND CODING PROTOCOL Many of the studies of mobile use by MSEs are qualitative, and do not report statistical findings. Even among quantitative studies, there is little agreement in terms of dependent and independent variables under scrutiny. Thus, a statistical metaanalysis would not be applicable [30]. Similarly, a method designed specifically for comparing ethnographies, such as reciprocal translation [31] would be unlikely to bridge qualitative and quantitative studies. The analysis draws instead on a systematic review methodology [32] to aggregate findings across the available studies. By using a standardized protocol, coding each individual study for the appearance or absence of certain assertions, the review assesses and parsimoniously represents what the research literature, in aggregate, suggests about mobile use by MSEs. The exercise relies on clearly articulated eligibility criteria to select studies and on standardized questions to evaluate them. These two levels of standardization, agreed upon before the formal review commenced, separates the exercise from a conventional listbased or thematic/narrative literature review.

Recently, studies have emerged that directly address how MSEs in the developing world are using mobiles rather than landlines or other ICTs. The studies are not as numerous as the enthusiasm in the popular press might suggest. They are a tiny fraction of the total literature on mobile use in the developing world [29]. They have emerged from different disciplines, and, as relative contemporaries, often do not cite each other. This section presents two studies representing distinct methodologies and conclusions, to provide an example of the range of available perspectives and to set the stage for the systematic review.


A. Selecting studies Papers were initially identified by online literature and database searches (using keyword combinations of mobile, cellular, microenterprise, MSE, etc.), and by snowball references from the bibliographies of studies already in hand. Next, these studies were assessed against a series of eligibility criteria: to be included, studies had to be specific to both mobiles and MSEs, report generalizable findings, and contain detailed primary data about mobile use in everyday conditions. What started out as a reasonably large body of studies was trimmed back significantly. In order to provide additional resources to other researchers, however, this section lists those excluded papers, along with the rationale for the decisions. To be included in the review, papers had to be specifically focused on mobile phones, which excluded some excellent research on landlines or payphones and MSEs [28, 33, 34]. The papers also had to be about MSEs, not SMEs. Papers that did not explicitly include sole proprietors and informal enterprises were excluded [35-37]. We made a more difficult decision to exclude papers that were not generalizable to a wide range of MSEs. An important line of research explores how many individuals earn livelihoods in the mobile business itself, by selling airtime, fixing handsets or operating village phones [38-40]. However, these studies treat mobiles as products and services, rather than enablers of general business processes. To fit into the evaluation protocol, papers had to offer sufficient details around the use of mobiles to illuminate their role in these business processes. A few surveys that were otherwise topically correct did not yield information of this kind [41], or blurred the lines between mobiles and other ICTs to the point where assertions about mobiles in particular were difficult to extract [42, 43]. Reviews without new primary data were excluded [44, 45]. When multiple papers drew on the same set of data [46, 47], only one paper was retained. Finally, the review focused on analyses of mobile use in everyday settings, rather than proposals for or evaluations of new pilot technologies [48-51] or programmatic interventions by NGOs [52]. The development of such technologies and programs is central to the ICTD field, but such initiatives yield different forms of evidence about mobile use than those that examine MSEs operating on their own. This limiting exercise forced a trade-off: the remaining papers clearly describe some element of the use of mobiles by MSEs in developing countries, but the population of such studies is relatively small. Thirteen papers and one book were retained. B. Evaluation protocol The process of developing the evaluation questions was iterative. It was based mostly on an initial reading of the documents by the researchers, while also integrating current narratives in the popular and practitioner literatures. An original goal was to code studies according to subcategories of MSEs (to discern differences in mobile use between traders

and producers, for example), but it became clear that the population of existing studies is too small to support that inquiry. The final protocol employed three distinct foci. First, it assessed the impact of mobile use on the internal process of an enterprise, using Porter’s value chain model [23], Fig. 1. The value chain comprises the activity inputs into a product or service: inbound logistics, operations (production), outbound logistics, marketing and sales, and after-sales service. The value chain also includes supporting functions: firm infrastructure, human resources, technology development (knowledge developed or owned by the enterprise), and procurement. Together these activities can create customer value in excess of the costs to provide it, yielding profit. Porter [23: 168], argues that information and communication technologies can be used to improve almost any of these primary and supporting activities. Although the value chain framework was developed with larger enterprises in mind, it can be applied to MSEs, since in small firms the same individual can carry out different business-related activities during the day. (Indeed, even in larger firms there is often not a perfect mapping between the activities and functions in the value chain and distinct people or departments). Both researchers coded individual papers for mentions of the mobile’s role in any of the primary or supporting functions. FIGURE 1: PORTER’S VALUE CHAIN

note: image released to public domain as per http://en.wikipedia.org/wiki/Image:ValueChain.PNG The second analysis used another Porter framework, the value system [23] ii , to offer a corresponding assessment of how mobile use influences the network of interdependencies and relationships external to the enterprise, including producers, traders, wholesalers, retailers and end-customers. An initial reading of the papers identified four categories of potential impacts. Some research stresses (a) the increased availability of information in the network; other studies stress (b) the entry of new actors, particularly buyers and sellers, into markets. Both factors tend to increase competition, but do


so in different ways. One focuses on the actors in the network, the other on the information those actors exchange. Nevertheless, the two changes do not necessarily move in tandem; it is one thing to assert that the same set of actors exchange information at lower cost and higher frequency, another to say that markets have expanded. This informationvs.-actors cut highlights this distinction. Two other categories of network impacts can be expressed as assertions that (c) mobiles help enterprises cut out middlemen and that (d) mobiles help individuals start new businesses. Unlike the value chain analysis (coded for affirmative mentions only), the value system analysis coded for both affirmations and negations of the four potential impacts. After the initial reading, we elected to track negations since some of the sources made a point of arguing against one or more of the assertions from the popular and practitioner literature. The third analysis explored how the impact of mobile use on small enterprises is or is not different from the impact of landline telephony on those same kinds of businesses [28, 33, 34]. Technological properties of mobile communication make it inherently more prone to adoption by MSEs than landlines: it is cheaper to build towers than lay cable, prepay accounts have no startup costs, and inexpensive/used handsets are readily available. However, in this case the third analysis focused narrowly on two differences in use rather than cost or access. First, studies were coded for mentions of mobility. Mobility is a crucial difference between mobiles and landlines—while landlines connect places to places, mobiles generally connect people to people, wherever they are and regardless of the time and situation. This mobility leads to increased individual addressability, and can change how people structure social and economic activity [53-55]. Most relevant to this analysis, mobility may enable the rise of roaming businesses, just-intime service and what Townsend [56] has called the “real time city”. And yet some evidence suggests that in the developing world many mobiles are purchased as substitutes for landlines, rather than complements to them [57]. Second, the nature of the mobile as a portable, personal device means it is particularly easy to use for both personal and business functions during the same day. Thus, studies of the role of mobiles in the lives of MSE operators are often different from studies of the role of the device in the businesses themselves. The analysis coded for studies that explore these social functions. Once the protocol was established, each researcher re-read the papers, coding them in isolation. We then compared our codes and resolved any discrepancies through discussion. The resulting codes are less prone to reflect the bias of a single reader. Of the 112 cells on the matrix requiring codes, 16 required discussion to resolve coding discrepancies between the two researchers.

V. RESULTS A. Enterprise value chain Most studies mention the core processes of marketing and sales [1, 2, 4, 6, 7, 9-13]. Analyses ranging from Jensen’s [10] model of searching for the best price for fish to Kamga’s [11] description of improvements to the local laundry services in Cote d’Ivoire asserts that mobiles help connect vendors and buyers, often at a distance and usually at lower cost than an in-person journey. Esselaar et al. [6] report results of a survey of SMEs, including 1/3 microenterprises, conducted across 13 countries. “Mobile phones are used more often for keeping in contact with customers and clients” (p 92). This is the highly visible, intuitive role of mobiles for small enterprises. The picture is sparser for other core processes within the value chain. Three studies mention inbound and outbound logistics [1, 9, 13], particularly Abraham [1], who details how fishermen can now use small supply boats (dispatched via mobile) to stay out fishing longer. Overå [13] describes how traders in Ghana can time harvests (inbound) and change the terms of delivery financing (outbound) because of the mobile. Operations receives two mentions, again by [1], who notes that fishermen use mobiles to coordinate the timing and location of when to drop nets and search for fish. Similarly, [9] describes how weavers call customers mid-process to revise plans for the garments they are creating. Only studies by Frempong et al. [7] and Molony [11] mention after-sales service. Molony describes how Tanzanian exporters of carved wood use the mobile to elicit feedback and built trust with buyers after (and ideally between) sales. In terms of crosscutting functions, five studies reference procurement [1, 6, 7, 9, 13] and address price search by buyers of inputs (or by traders). There is little evidence to date for the mobile’s role in transforming the proprietary technology, infrastructure or HR functions of MSEs, perhaps because these enterprises are too small to invest in these assets. Although studies outside the review [36, 58] provide anecdotes of small employers giving mobiles to employees, this infrastructure function is of limited utility for tiny firms and sole-proprietorships. B. Industry value system The second analysis turns the lens outside the enterprise, towards its location in a network of relationships. The most common finding links mobile use to an increase in the flow of information between actors in the value system [1, 2, 6, 7, 914]. The two primary sub-themes are more frequent or wideranging exchanges of price information [1, 2, 13], and a more generalized discussion of increased communication with customers [6, 12, 14]. These findings are reflections of the frequent references to marketing and sales and procurement activities in the previous value chain analysis. Reference [9] mentions an increase in the completeness of the information, but notes that they saw no increase in quality



Enterprise Value Chain Studies
Core Processes
[1] R. Abraham, “Mobile phones and economic development: evidence from the fishing industry in India,” [2] J. C. Aker, “Does digital divide or provide? The impact of cell phones on grain markets in Niger” [3] J. Donner, “Microentrepreneurs and mobiles: An exploration of the uses of mobile phones by small business owners in Rwanda” [4] J. Donner, “The use of mobile phones by microentrepreneurs in Kigali, Rwanda: Changes to social and business networks” [5] J. Donner, “Customer acquisition among small and informal businesses in urban India: Comparing face to face, interpersonal, and mediated channels” [6] S. Esselaar, C. Stork, A. Ndiwalana, and M. Deen-Swarra, “ICT usage and its impact on profitability of SMEs in 13 African Countries” [7] G. Frempong, G. Essegbey, and E. Tetteh, “Survey on the use of mobile telephones for micro and small business development: The case of Ghana,” [8] H. Horst and D. Miller, “The Cell Phone: An Anthropology of Communication.” [9] A. Jagun, R. Heeks, and J. Whalley, “The Impact of Mobile Telephony on Developing Country Micro-Enterprise: A Nigerian Case Study” [10] R. Jensen, “The Digital Provide: Information (Technology), Market Performance, and Welfare in the South Indian Fisheries Sector” [11] O. Kamga, “Mobile phone in Cote d'Ivoire: uses and self-fulfillment” [12] T. Molony, “‘I don't trust the phone; it always lies’: Trust and information and communication technologies in Tanzanian micro- and small enterprises” [13] R. Overå, “Networks, distance, and trust: Telecommunications Development and changing trading practices in Ghana” [14] J. Samuel, N. Shah, and W. Hadingham, “Mobile Communications in South Africa, Tanzania, and Egypt: Results from Community and Business Surveys” Inbound & Outbound Logistics, Operations, Marketing & Sales. Marketing & Sales

Industry Value System
Add Buyers/ Sellers


Add Cross-cutting InformFunctions ation
Procurement Yes

Bypass Middlemen

Start businesses

















Marketing & Sales
















Marketing & Sales








Marketing & Sales; Service








-Inbound & Outbound Logistics, Operations, Marketing & Sales Marketing & Sales






















Marketing & Sales Marketing & Sales; Service Inbound Logistics, Outbound Logistics, Marketing & Sales --






























While it is one thing to say that mobile use accelerates the flow of information in existing value systems, it is another to say that mobile use brings new customers or suppliers into the market. This is the first element in the systematic review in which there is some disagreement between the primary studies. Numerous studies present evidence that mobile use expands markets by allowing MSEs to reach new customers [1, 2, 4, 7, 9, 10, 13, 14]. Of the Keralan fishermen, [10: 891] explains “while almost all sales before mobile phones were conducted via beach actions, fishermen with phones, often carrying lists with the numbers of dozens or even hundreds of potential buyers, would typically call several buyers in different markets before deciding where to sell their catch”. Similarly, Aker finds that “Grain traders in markets with cell phone coverage search over a greater number of markets, have more contacts and sell in more markets. This underscores the fact that the primary mechanism by which cell phones affect market efficiency is a reduction in search costs and hence transaction costs.” [2: 4-5] In the study, mobile use lowered price dispersion by 21%, increased profits by 29%. And yet two other studies specifically argue that the phones have done little to introduce new buyers. Donner [5] finds that MSEs in urban India are much more likely to recruit customers via face-to-face channels, rather than via a phone call. [12] argues that mobiles help accelerate and strengthen trusting relationships but only among parties that have already established a face-to-face bond. Two other general assertions about the impact of mobile use on MSE value systems receive less support from the studies. None of the studies asserts that mobiles help MSEs bypass middlemen. Indeed three of the papers focus specifically on middlemen, wholesalers or traders as enterprises, [2, 9, 13] describing how mobiles allow them to perform their roles more effectively. Another specifically emphasizes how producers work with existing middlemen in their industries, rather than routing around them. Rather than radically restructuring these marketplaces, Molony argues, “mobile phones can be seen as a facilitating technology for existing, trust-based relationships” [12: 78] Similarly, there is relatively little evidence for the assertion that mobiles help people start new businesses. Only Samuel et al. [14] make this case, reporting that among a sample of MSEs in Egypt and South Africa, 26%-29% of businesses attributed their start to the availability of the mobile. Taking the opposite position, Horst and Miller [8: 164] argue that despite some isolated examples to the contrary (taxi drivers and musicians), “there is no new spirit of enterprise based on either the cell phone or the internet” among the Jamaican households in their study. Nevertheless, they argue that despite a dearth of new enterprises, the mobile is essential to the economic survival of those households. By allowing individuals to leverage broad networks of informal social and financial support through a process Jamaican mobile users call “link up,” “the phone is not central to making money, but is vital to getting money.” [8: 165] In sum, in value systems where mobile telephony is

introduced, there is more evidence for changes in degree (more information, more customers) than for changes in structure (new channels, new businesses). C. On attributes of the mobile vs. the landline Roughly half of the studies described use cases that take advantage of mobility. Clearly, fishermen take advantage of wireless telecommunications [1, 10] to place and receive calls while on the water. This is not only an advantage for determining which markets to target, but [1] points out that it also enables fishermen to feel safer while at sea. Traders [2, 13] use the mobile to be individually addressable wherever they are. Reference [11] illustrates the responsiveness of businesses that can serve the customer, 24 hours a day, while [13] portrays “availability as comparative advantage”, and argues that this more frequent interaction builds trust between suppliers and customers. Given that MSE operators often carry their mobiles throughout the day and into the evening, a blending of mediated communication for social and instrumental purposes often occurs. While some of the papers in the review focus exclusively on the business functions, others [1, 3-6, 8, 11-14] illustrate this blurring. Blurring occurs at the aggregate level, —a survey by Donner [4] found that roughly 1/3 of calls made by MSE owners in Rwanda were business-related. It also occurs within individual calls—non-business (“chit-chat”) exchanges increase trust between clients and customers [12, 13]. Finally [8] describes the “link up” process in Jamaica, in which individuals retain a roster of numbers of friends, family and acquaintances that can be tapped periodically for loans or small cash gift transfers. This process intermingles social and economic functions of mobile use. VI. DISCUSSION This paper offers a systematic review of the current research on the impact of mobile use on MSEs, applying both an internal (value chain) and external (value system) perspective. The review finds a pattern of evidence suggesting that mobiles increase the information available to MSEs. Some [2, 10] provide quantitative evidence for how this information translates into reduced price variability and higher profits per actor. The current studies suggest mobiles are most useful for streamlining marketing and sales (downstream) and procurement (upstream) with existing business contacts. In some cases, studies suggest that mobile use expands the size of markets by bringing a larger number of buyers and sellers into the marketplace. However not all studies found evidence that new customers were acquired. Far fewer studies present evidence that mobiles enable the creation of new businesses, or that mobile use re-organizes value systems to allow producers to bypass middlemen. Indeed, middlemen are positioned to take advantage of mobiles themselves. To summarize, the review of the evidence offered across the thirteen studies suggests that within the MSE sector, benefits of mobile use accrue mostly (but not exclusively) to existing enterprises, in ways that amplify and accelerate


material and informational flows, rather than fundamentally transform them. This summary does not diminish the positive utility of mobiles to MSEs, but it places that utility in context and in doing so echoes assertions by Castells [54] and Harper [59]. Mobile use by MSEs can be understood as an extension of the information society, not its restructuring [60]. A second theme in the analysis involves mobiles as complements or substitutes for landlines. Evidence here remains mixed; while some case studies highlight enterprises that take advantage of “availability as comparative advantage” [13], there is insufficient data to determine whether these are isolated cases or representations of a more universal condition. The majority of MSEs may take greater advantage of place-to-place connectivity, than mobility, but this point merits further study. A. Generalization and segmentation Clearly, mobiles offer distinct benefits to MSEs— everything from more accurate price information and access to new customers to better after-sales service and procurement, from increased responsiveness to the opportunity to build trust at a distance. However, it is currently difficult to determine whether the various impacts and benefits enumerated in this review accrue equally to all MSEs. Looking across a diversity of MSEs, across nations, industries, and different locations in value systems, the current research points toward a multiplicity of intertwined and sometimes contradictory impacts of mobile use, rather than the universal and rather uncomplicated benefits which have characterized the popular rhetoric. To guide future policy or institutional interventions, it would be helpful to deploy future research against a set of open questions: Which kinds of MSEs are gaining the most return on mobile use? Which kinds (and what proportion) of MSEs are poised to find new customers and expand their markets, and not simply be more responsive to the ones they already have? Which kinds (and what proportion) are unlikely to reap any benefit from mobiles, or actually may be threatened by changes in mobile use elsewhere in their value systems? These questions remain open because most studies to date have been either sector-specific explorations or broad aggregate surveys; few studies specifically identify differences in mobile use or impact among subpopulations of MSEs. By contrast, recent studies of mobile use by farmers are identifying factors which differentiate between subgroups, for example, between growers of perishable and nonperishable crops [61], by distance from local markets [61], or according to different levels of infrastructural constraint [62]. An important path for further study would apply similar comparative analyses to assess and predict the impact of mobile use by different classes of MSEs. These are hefty quantitative tasks. Future designs will require increased attention to the factors that distinguish subgroups of MSEs as well as careful measurement of desirable outcomes such as productivity, market participation, or revenue growth. However, this review helps to identify a

range of variables for both the independent and dependant sides of such analyses. Indeed, a quantification of mobile use by subpopulations of MSEs may begin to close the gap between micro-level case studies and research on the impact of mobiles on macroeconomic growth [63]. B. Enterprises, livelihoods or lives? The conclusions of this review naturally depend on the methods and theoretical frameworks employed at the primary and secondary stages of analysis. Ethnographies such as [11] and [9] tended to discuss a broader range of uses and impacts than focused quantitative tests [2, 5, 10]. Similarly, the lenses chosen for this systematic review highlight some dimensions of MSE use over others. For example, a livelihoods framework [64], instead of enterprise-specific frameworks from Porter [23] would have emphasized different patterns. The dichotomous treatment of new vs. existing businesses has its limitations. Reference [9] describes the evolution of some weavers to “coordinator weavers,” suggesting a transformation in the structural location and internal processes of some enterprises, but this was an exception among the studies, most of which looked only to existing enterprises for their sample. The methods used by [8] can identify occupational multiplicity—holding down more than one job at once—in a way that studies focused specifically on existing MSEs cannot do. Similarly, [7] is able to assert that few households start new enterprises thanks to mobile use only because households rather than MSEs are its primary units of analysis. These examples reveal that insights about use of mobiles in MSEs can come from studies that focus not exclusively on enterprises, but rather on the individuals who manage them. Focusing on individuals also allows for increased linkage to research on social uses of the mobile. C. New applications on the mobile platform Studies have documented how mobiles can enable information search and improve communication between MSEs, customers and suppliers. However, there has been little evidence to date that suggests mobiles are being used for information storage or processing. As was the case with landlines [27], MSEs value voice calls more than any other function on the mobile, and use the calls to augment, rather than replace, face-to-face ties. As [6: 99] explains, mobiles “cannot be used to track inventory, provide cash flow and income statements, or even more basically, produce formal letters, marketing campaigns, or brochures”. Recently, however, various systems have appeared that go beyond the voice and peer-to-peer texting functions on the handset. These make the handset approximate a PC (with processing happening on the handheld), or as the client in a client-server model, with primary processing happening elsewhere on the network. These latter models take advantage of everything from basic SMS [65] to voice to full-blown mobile internet browsing experiences. Two of the more promising applications are distributed marketplaces, such as


Cell Bazaar, Manobi, and Tradenet, and mobilebanking/mobile-payments initiatives such as Kenya’s popular M-PESA [66]. In addition, a variety of promising pilots are underway, such as [48-51], which promise to yield further functionality, for example, in supply chain management. Full evaluations of the use and impact of these services on MSEs are not yet available, and are urgently needed. This baseline analysis can help clarify which impacts these systems may be having. Findings that suggest, for example, that mobile trading platforms help MSEs bypass middlemen, would be even more impressive given this synthesis’ conclusion that there is limited evidence that basic voice calls can have this effect. Similarly, findings that suggest MSEs are using m-banking or m-payments applications to transform credit relationships or otherwise change the procurement and sales functions could be assessed more accurately against the baseline of the voice-based behaviors found so far. VII. CONCLUSION This paper has summarized fourteen primary research studies examining mobile use by MSEs; the research generally concurs with conventional wisdom—mobile use helps many MSEs become more productive, particularly but not exclusively via improvements to sales and marketing and procurement processes. That said, the review suggests that not all enterprises will prosper from increased access to telecommunications, and among those that do, their uses of mobiles will vary across industries and positions in value systems. As mentioned above, current evidence suggests that the benefits of mobile use accrue mostly (but not exclusively) to existing enterprises, in ways that amplify and accelerate material and informational flows, rather than fundamentally transform them. On balance, MSEs are likely to remain unproductive relative to larger enterprises [17]. However, the improvements to productivity associated with mobile use do seem to be improving the livelihoods of many individuals in the MSE sector. The results of this review are helpful to the ICTD research community in at least three ways. First, by disaggregating and identifying distinct impacts of mobile use, the review provides a more nuanced and more accurate representation of the value of mobile use to MSEs than was previously available. Second, the review identifies a skew (in both sample and implied impacts) towards existing enterprises that should be noted by policymakers who may expect mobiles to create new businesses and new employment. Finally, the review identifies two priorities for future research: a segmentation and further quantification of impacts by subsectors of MSEs, and an assessment of the use of new non-voice advanced mobile services (such as mobile banking and mobile marketplaces) by MSEs REFERENCES
[1] R. Abraham, “Mobile phones and economic development: Evidence from the fishing industry in India,” Proceedings of the







[8] [9]

[10] [11]


[13] [14]

[15] [16] [17] [18] [19] [20] [21] [22]

International Conference on Information and Communication Technologies and Development, (ICTD 2006). Berkeley, CA: IEEE, pp. 48-56, 2006. J. C. Aker, “Does digital divide or provide? The impact of cell phones on grain markets in Niger,” BREAD Working Papers. vol. 177, Durham, NC: Bureau for Research and Economic Analysis of Development, 2008. J. Donner, “Microentrepreneurs and mobiles: An exploration of the uses of mobile phones by small business owners in Rwanda,” Information Technologies and International Development, vol. 2, pp. 1-21, 2004. ___, “The use of mobile phones by microentrepreneurs in Kigali, Rwanda: Changes to social and business networks” Information Technologies and International Development, vol. 3, pp. 3-19, 2006. ___, “Customer acquisition among small and informal businesses in urban India: Comparing face to face, interpersonal, and mediated channels,” The Electronic Journal of Information Systems in Developing Countries, vol. 32, pp. 1-16, 2007. S. Esselaar, C. Stork, A. Ndiwalana, and M. Deen-Swarra, “ICT usage and its impact on profitability of SMEs in 13 African countries,” Information Technologies and International Development, vol. 4, pp. 87-100, 2007. G. Frempong, G. Essegbey, and E. Tetteh, “Survey on the use of mobile telephones for micro and small business development: The case of Ghana,” Institutes of the Council for Scientific and Industrial Research, Science and Technology Policy Research (STEPRI), Accra, 2007. H. Horst and D. Miller, The cell phone: An anthropology of communication. Oxford: Berg, 2006. A. Jagun, R. Heeks, and J. Whalley, “The impact of mobile telephony on developing country micro-enterprise: A Nigerian case study,” Information Technologies and International Development, vol. 4, pp. 47-65, 2008. R. Jensen, “The digital provide: Information (technology), market performance, and welfare in the South Indian fisheries sector,” Quarterly Journal of Economics, vol. 122, pp. 879-924, 2007. O. Kamga, “Mobile phone in Cote d'Ivoire: Uses and selffulfillment,” Proceedings of the International Conference on Information and Communication Technologies and Development, (ICTD 2006). Berkeley, CA, pp. 184-192, 2006. T. Molony, “‘I don't trust the phone; it always lies’: Trust and information and communication technologies in Tanzanian microand small enterprises,” Information Technologies and International Development, vol. 3, pp. 67-83, 2006. R. Overå, “Networks, distance, and trust: Telecommunications development and changing trading practices in Ghana,” World Development, vol. 34, pp. 1301-1315, 2006. J. Samuel, N. Shah, and W. Hadingham, “Mobile communications in South Africa, Tanzania, and Egypt: Results from community and business surveys,” in Africa: the impact of mobile phones. vol. 2007, Vodafone, Ed. London: Moving the Debate Forward: The Vodafone Policy Paper Series #3, 2005, pp. 44-52. International Labor Organization, “Key indicators of the labour market,” International Labor Organization, Geneva 2003. D. C. Mead and C. Leidholm, “The dynamics of micro and small enterprises in developing countries,” World Development, vol. 26, pp. 61-74, 1998. R. la Porta and A. Shleifer, “The unofficial economy and economic development,” Brookings Institute, Washington, DC 2008. D. Snodgrass, “Inventory and analysis of donor-sponsored MSE development programs,” Development Alternatives Inc., Washington, DC 2005. Economist, “Calling an end to poverty,” The Economist. vol. 376, 2005, pp. 51-52. S. LaFraniere, “Cellphones catapult rural Africa to 21st century,” in The New York Times New York, p. 1, 2005. USAID, “Using cellular phones in Uganda for rural income generation and more,” Issue 7 ed. vol. 2006: DOT-COM Alliance E-Newsletter, 2004. A. Bhavnani, R. Won-Wai Chiu, S. Janakiram, and P. Silarszky, “The role of mobile phones in sustainable rural poverty reduction,”

World Bank Global Information and Communications Department, Washington, DC 15 June 2008. M. E. Porter, Competitive advantage: Creating and sustaining superior performance. New York: Free Press, 1985. D. C. Mead and C. Morrison, “The informal sector elephant,” World Development, vol. 24, pp. 1611-1619, 1998. R. Duncombe and R. Heeks, “Enterprise development and information and communications in developing countries: Supporting ‘ICT flyers’,” Manchester: IDPM, University of Manchester, 2001. R. J. Saunders, J. J. Warford, and B. Wellenieus, Telecommunications and economic development, 2 ed. Baltimore, MD: Johns Hopkins University Press, 1994. R. Duncombe and R. Heeks, “Information and communication technologies and small enterprise in Africa: Findings from Botswana." vol. 2006 Manchester, UK: Institute for Development Policy and Management, University of Manchester, 1999. ___, “Enterprise across the digital divide: Information systems and rural microenterprise in Botswana,” Journal of International Development, vol. 14, pp. 61-74, 2002. J. Donner, “Research approaches to mobile use in the developing world: A review of the literature,” The Information Society, vol. 24, pp. 140-159, 2008. R. J. Light and D. B. Pillemer, Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press, 1984. G. W. Noblit and R. D. Hare, Meta-ethnography: Synthesizing qualitative studies. Newbury Park, CA: Sage, 1988. J. H. Littell, J. Corcoran, and V. Pillai, Systematic reviews and meta-analysis. New York: Oxford University Press, 2008. D. Souter, N. Scott, C. Garforth, R. Jain, O. Mascararenhas, and K. McKemey, “The economic impact of telecommunications on rural livelihoods and poverty reduction: A study or rural communities in India (Gujarat), Mozambique, and Tanzania,” Commonwealth Telecommunications Organisation for UK Department for International Development, 2005. R. Duncombe and R. Heeks, “Information, ICTs and small enterprise: Findings from Botswana,” Small Scale Enterprises in Developing and Transitional Economies, H. Katrak and R. Strange, Eds. New York: Palgrave, 2002, pp. 285-304. J. O. Adeoti and A. I. Adeoti, “Easing the burden of fixed telephone lines on small-scale entrepreneurs in Nigeria: GSM lines to the rescue,” Telematics and Informatics, vol. 25, pp. 1-18, 2008. R. A. Boadi, R. Boateng, R. Hinson, and R. A. Opoku, “Preliminary insights into m-commerce adoption in Ghana,” Information Development, vol. 23, pp. 253-265, 2007. W. Mei and L. H. Yun, “Perpetual contact for business: Mobile phones and Fujian entrepreneurs,” Proceedings of ANZCA08: Power and Place, Wellington, NZ, 2008. S. Aminuzzaman, H. Baldersheim, and I. Jamil, “Talking back: Empowerment and mobile phones in rural Bangladesh: A study of the village pay phone of Grameen bank,” Contemporary South Asia, vol. 12, pp. 327-348, 2003. H. Ilahiane and J. Sherry, “Joutia: Street vendor entrepreneurship and the informal economy of information and communication technologies in morocco,” The Journal of North African Studies, vol. 13, pp. 243-255, 2008. J. Lugo and T. Sampson, “E-informality in Venezuela: The 'other path' of technology,” Bulletin of Latin American Research, vol. 27, pp. 102-118, 2008. B. F. Chogi, “The impact of mobile phone technologies on medium and small enterprises/jua kali (MSEs),” Proceedings of CPRsouth2 Chennai, 2007. R. O. Opiyo and O. A. K'Akumu, “ICT application in the informal sector: The case of the Kariokor market MSE cluster in Nairobi,” Urban Forum, vol. 17, pp. 241-261, 2006. E. D. Moyi, “Networks, information and small enterprises: New technologies and the ambiguity of empowerment,” Information Technology for Development, vol. 10, pp. 221-232, 2003. K. O. Davis, Cosmas. , “ICTs as appropriate technologies for African development” IFC/FT First Annual Essay Competition: Business & Development: The Private Path to Prosperity, Washington, DC: Financial Times Limited / International Finance Corporation (IFC), pp. 43-54, 2007. A. T. Rashid and L. Elder, “Mobile phones and development: An analysis of IDRC-supported projects,” Electronic Journal of Information Systems in Developing Countries, vol. 36, pp. 1-16, 2009. T. Molony, “Running out of credit: The limitations of mobile telephony in a Tanzanian agricultural marketing system,” Journal of Modern African Studies, vol. 46, pp. 1-22, 2008. ___, “The role of mobile phones in Tanzania’s informal construction sector: The case of Dar es Salaam” Urban Forum, vol. 19, pp. 175-186, June 2008. P. S. Javid and T. S. Parikh, “Augmenting rural supply chains with a location-enhanced mobile information system,” Proceedings of the International Conference on Information and Communication Technologies and Development, (ICTD 2006). Berkeley, CA pp. 110-119, 2006. A. Kumar, N. Rajput, S. Agarwal, D. Chakraborty, and A. A. Nanavati, “Organizing the unorganized - employing IT to empower the under-privileged,” Proceedings of the 17th international conference on World Wide Web, Beijing, pp. 935-944, 2008. D. Chakraborty, K. Dasgupta, S. Mittal, A. Misra, A. Gupta, E. Newmark, and C. L. Oberle, “Businessfinder: Harnessing presence to enable live yellow pages for small, medium and micro mobile businesses,” IEEE Communications Magazine, vol. 45, pp. 144151, 2007. S. Biswas and S. Roy, “A shared ICT infrastructure for Indian SME clusters,” in 2nd IEEE/ACM International Conference on Information and Communication Technologies and Development (ICTD2007), Bangalore, pp. 221-230, 2007. Food India, “Inter-city marketing network for women microentrepreneurs using cell phone,” Food India 2002. J. E. Katz and M. Aakhus, “Perpetual contact: Mobile communication, private talk, public performance,” Cambridge, UK: Cambridge University Press, 2002. M. Castells, J. L. Qiu, M. Fernández-Ardèvol, and A. Sey, Mobile communication and society: A global perspective (information revolution and global politics). Cambridge, MA: MIT Press, 2007. C. Licoppe, “'connected' presence: The emergence of a new repertoire for managing social relationships in a changing communication technoscape,” Environment and Planning D: Society and Space, vol. 22, pp. 135-156, 2004. A. M. Townsend, “Life in the real-time city: Mobile telephones and urban metabolism,” Journal of Urban Technology, vol. 7, pp. 85-104, 2000. J. Hamilton, “Are main lines and mobile phones substitutes or complements? Evidence from Africa,” Telecommunications Policy, vol. 27, pp. 109-133, 2003. J. Donner, “The social and economic implications of mobile telephony in Rwanda: An ownership/access typology,” in Thumb culture: The meaning of mobile phones for society, P. Glotz, S. Bertschi, and C. Locke, Eds. Bielefeld, Germany: Transcript Verlag, 2005, pp. 37-52. R. Harper, “Are mobiles good or bad for society?,” in Mobile democracy: Essays on society, self and politics, K. Nyíri, Ed. Budapest, Hungary: Passagen Verlag, pp. 185-214, 2003. J. Donner, “Shrinking fourth world? Mobiles, development, and inclusion,” in Handbook of mobile communication studies, J. Katz, Ed. Cambridge, MA: MIT Press,2008, pp. 29-42. M. Muto and T. Yamano, “The impact of mobile phone coverage expansion on market participation: Panel data evidence from Uganda,” Japan Bank for International Cooperation, Tokyo 2008. S. Gandhi, S. Mittal, and G. Tripathi, “The impact of mobiles on agricultural productivity,” in India: The Impact of Mobile Phones, The Vodafone Policy Paper Series #9 London: Vodafone, 2009, pp. 21-33. L. Waverman, M. Meschi, and M. Fuss, “The impact of telecoms on economic growth in developing nations." vol. 2007 London: Moving the Debate Forward: The Vodafone Policy Paper Series #3, 2005, pp. 10-23. R. Duncombe, “Using the livelihoods framework to analyze ICT applications for poverty reduction through microenterprise,”

[23] [24] [25]


[46] [47] [48]

[26] [27]

[28] [29] [30] [31] [32] [33]




[52] [53] [54] [55]


[35] [36] [37] [38]

[56] [57] [58]


[59] [60] [61] [62]

[40] [41] [42] [43] [44]



Information Technologies and International Development, vol. 3, pp. 81-100, 2007. R. Veeraraghavan, N. Yasodhar, and K. Toyama, “Warana unwired: Mobile phones replacing PCs in a rural sugarcane cooperative,” Proceedings of the 2nd IEEE/ACM International Conference on Information and Communication Technologies and Development (ICTD2007), Bangalore, pp. 89-98, 2007. O. Morawczynski and G. Miscione, “Examining trust in mobile banking transactions in Kenya: The case of m-PESA in Kenya,” in IFIP WG 9.4-University of Pretoria Joint Workshop, Pretoria, South Africa, 2008



The analysis includes two studies of fishermen but excludes farmers ii The term ‘value system’ is sometimes used interchangeably with ‘industry value chain’. This review uses the Porter nomenclature to distinguish between the intra- and extraenterprise systems



An Evaluation of the use of ICT within Primary Education in Malawi
David Hollow and Paola Masperi
the authority of the Ministry of Education, Science and Technology (MoEST). Funding for the initiative was provided by a British company who also developed the educational software for the portable devices. II. CONTEXT A. Education in Malawi Free primary education was introduced in Malawi in 1994 as a result of the commitment made by the country at the World Conference on Education for All (EFA) in 1990. Widespread agreement exists regarding the vital place of education within poverty reduction efforts, capacity building and growth strategies of developing countries. In Malawi, numerous policy, budgetary and multilevel commitments have contributed to significant progress in the delivery of primary education since 1994. This includes enrolment figures rising from 1.9 million to 3.2 million, the construction of 1,000 new classrooms and roll out of the new curriculum to 5,500 schools [1]. However, Malawi is one of the poorest countries in the world, with a GNI per capita of $230. Life expectancy at birth is 48 years and 63% of the population live on less than $2 per day [2]. Within such a context of extreme poverty, the rapid advancements in access to free schooling have put the national education system under considerable strain. The increase in enrolment has led to significant pressure upon primary schools in Malawi, most notably in regard to increasing class sizes, a lack of fully qualified teachers, limited teaching materials and inadequate infrastructure. It is estimated that an extra 8,000 teachers are currently required to meet the MoEST target pupil to teacher ratio of 1:60 [3]. In addition, and despite national net enrolment levels of 91%, national drop out rates remain high with only 44% of those that enroll in Standard 1 completing Standard 5 and less than 30% reaching Standard 8, the final year of primary education [4]. Linked to this and demonstrating the ongoing challenge in regard to teaching capabilities and educational outcomes is a declining performance in national examinations with a failure rate of over 40% [1]. The importance of ensuring good quality education alongside improved access [5] has been long recognized in Malawi and in 1999 it was decided that a comprehensive

Abstract— The paper demonstrates how appropriate technology, when combined with quality curriculum-based content, has the potential to have a positive impact on primary education within developing countries. It documents an evaluation of portable learning technology from the Ministry of Education, Science and Technology in Malawi, providing theoretical background and educational context before detailing the methodology and findings of the study. Significant impacts from the introduction of the program were increased school attendance, reduced dropout rates and improved student and teacher enthusiasm. Alongside this, the audio and video material and interactive learning techniques offered potential pedagogical benefits in combining learner-centered and outcome-based activities with continuous assessment, helping children to retain information more effectively and record higher test scores. Major challenges were also identified regarding implementation and sustainability. These centered around the need for rigorous teacher training, classroom integration, appropriate deployment, maintenance, sustained impact and overall cost-effectiveness. In closing, the paper emphasizes the need for such programs to be driven by educational concerns and recognizes the similar challenges faced in many related initiatives. Index Terms— Education, Evaluation, Malawi, Technology



N setting the context for the subsequent evaluation the paper begins with an overview of two distinct spheres, firstly the current education context in Malawi and secondly the potential role of technology within education. These two seemingly disparate themes are brought together through the case study of the Interactive Learning Program. An overview of the program is provided, followed by an explanation of the methodological approach employed in the monitoring and evaluation exercise. The findings are then categorized into four sections of analysis which inform the concluding comments and recommendations. The Interactive Learning Program in Malawi comes under

Manuscript received September 21, 2008. David Hollow is a PhD candidate with the ICT4D Collective in the Geography Department at Royal Holloway, University of London. e-mail: d.m.hollow@rhul.ac.uk Paola Masperi is an independent development consultant, based in London. e-mail: paola@mayamiko.org


reform of the curriculum was required due to the developments that had occurred since 1994 [3]. The relevance of topics such as democracy, human rights, gender and HIV/AIDS were increasingly acknowledged and these have been systematically incorporated into the new primary school curriculum in the form of life skills education. B. Technology in Education There is significant attention focused upon the potential for ICT to assist in leapfrogging educational challenges within developing nations [6] [7] [8], with much enthusiasm for a possible technology-enabled ‘breakthrough in learning’ [9]. However, much debate surrounds the question of whether the infusion of technology into education has actually instigated more than incremental changes to the field. Proponents assert that the last decade have resulted in the emergence of a new landscape for education across the globe [10] with technology positively effecting student motivation [11] and, when initiatives are implemented with fidelity, leading to a significant increase in learning [12]. This is challenged by those who argue that achieving structural technological change in schools takes much longer than anticipated [13] with no substantial evidence that the introduction of technology has yet caused any fundamental change in either a developed or developing context [14]. Wagner [15] notes a wide variety of outcomes from ICT for education projects in developing countries, with significant negative impacts including the reinforcing of dependencies, imposition without community involvement and collapse due to lack of funding or political commitment. Within this contested environment, the range of technologies theoretically available for deployment within developing country educational contexts is rapidly expanding, with a transition towards increasingly portable, powerful and adaptable tools [16]. The new technologies available have potential to mark a significant transition beyond conceptualizing e-learning through conventional static computer laboratories [17], emphasising instead the place of anytime, anywhere computing [16] and mobile learning [18]. Such a shift creates opportunity for major changes in the application of educational technology, with key potential benefits noted as increased enthusiasm, cooperation, communication and student ownership [19]. Despite the potential benefits of learning technologies [20] it is important to recognize that not all solutions which have contributed to educational advancements in the developed world can simply be transferred to a developing world context. In order for the potential of ICTs to be realized in improving the quality of education it is important they are applied with cultural understanding, local knowledge and sensitivity [11] [21]. Indeed, ‘effective use of ICTs must be tied to the needs of developing countries and challenge the one size fits all approach of many programmes’ [19;7]. A recognition of this fact emphasises the need for primary focus on appropriate software development linked to curriculum-

based content and sustained classroom integration. III. INTERACTIVE LEARNING PROGRAMME Having outlined the overall context of education within Malawi and the theoretical role of technology within education, attention now turns to the Interactive Learning Program. It considers whether the initiative constitutes a suitable application of portable technology to support the provision of basic education within the country. To begin, the specifics of the program are explained, including the background, technology and pedagogical rationale. A. Background to Program The Interactive Learning Program was introduced in 2006 and following initial positive feedback the MoEST requested, in early 2007, that the initiative be scaled up to incorporate 50 primary schools. The schools that the MoEST selected for participation ranged from Karonga District in the north through to the southernmost districts of Mulanje and Phalombe. A total of 520 custom-made handheld interactive learning aids were distributed to the participating schools. This totaled ten devices for each school, except for two test schools which received 20 devices for the purposes of targeted monitoring and evaluation. B. Technology Utilized The device in question is a handheld interactive learning aid (from herein referred to as the learning machine or gadget as named by the participating children), slightly larger than a mobile phone and able to play video and audio through either a loudspeaker or headphones. Positioned next to the screen are a selection of buttons which can be pressed in response to questions asked in the lesson being watched. The device has an in-built rechargeable lithium battery with power for between four and six hours of continuous play. A total of 25 lessons in Chichewa and 40 in English are preinstalled and stored on two gigabytes of internal Flash memory, leaving additional room for newly developed lessons to be incorporated. The lessons are designed for use by children in Standard 3, 4 and 5, each lasting approximately 20 minutes and covering a range of curriculum based subjects such as General Studies, Social Studies, Science, Mathematics, Geography, Life Skills and English. With less than 5% of primary schools across Malawi estimated as having access to a reliable supply of mains electricity [3] the initiative was dependent upon solar charging systems. Each of the project schools were equipped with a 20Watt solar panel connected to a deep-cycle sealed battery, generating enough energy to power 20 of the learning machines. Teachers were given training regarding appropriate charging and usage, and instructed to leave the devices to recharge overnight once every two days.


C. Pedagogy The digital content on the learning machines is designed in order to enable and promote interactive and outcome-based learning, linked to the Malawi national curriculum and actively supporting the government Primary Curriculum Assessment Reform [22]. The lessons are designed to be user friendly, with the audio-visual content accessible to illiterate or semi-literate users. Instructions from lip synchronized cartoon characters explain the functions of the various buttons on the learning machine. Once a lesson title has been selected the methodology becomes interactive and learner-centered. Having listened to brief teaching points the learners are tested on their understanding of the information supplied through the posing of multiple-choice questions answered by pressing buttons on the device. At the end of the lesson learners are given opportunity to test what they have learnt through undertaking an overall quiz of ten questions where they are congratulated according to the score attained. This outcomebased approach provides incentive for the children to work conscientiously and allows the teacher to monitor progress. The decision to include lessons in both Chichewa and English was taken on the basis that early learning content is most effective when communicated in the vernacular language [23] [24]. The dual language approach equips children for the challenging transition from Chichewa to English as the official medium of school instruction which takes place in Standard 5. The most conducive learning environment for using the devices is with groups of between four and six students. In this context team work, group participation and the development of leadership skills can each be encouraged. IV. METHODOLOGICAL APPROACH In regard to the overall context of technology within education, Kozma [25;21] concludes after assessing a wide variety of studies that there is ‘no consistent relationship between the mere availability or use of ICT and student learning’. Beyond such generic assertions, there are significant knowledge gaps remaining regarding what kind of initiative works and what does not. As InfoDev [11;5] summarize, ‘despite thousands of impact studies, the impact of ICT use on student achievement remains difficult to measure and open to much reasonable debate’. In light of such observations all partners in the Interactive Learning Program initiative recognized the necessity for a comprehensive monitoring and evaluation exercise, the key objectives of which were to; • assess the impact of the program on primary education • assess the feasibility of the program in each test school • identify program weaknesses for future refinement • develop both teacher and organizational capacity • provide feedback to partners regarding suitability of scaleup and sustainability A common theme from ICT-enhanced education programs within developing countries is the marginalized place of

monitoring and evaluation. Where undertaken, there is often strong emphasis placed on hardware-based input indicators [25] rather than more complex learning outcomes. In order to avoid this simplification and demonstrate an alterative approach, a rigorous monitoring, evaluation and impact assessment structure was built into the program throughout implementation. This involved three field visits from the monitoring and evaluation team in September 2007, November 2007 and March 2008. In addition to being marginalized, monitoring and evaluation of ICT for education programs has rarely embraced creative processes of assessment [26]. Recognizing this, a systemsbased approach was utilized in designing the methodology [27] [28] [29]. This involved adopting a mixed- and multimethod approach which focused on process and aimed to engage with multiple stakeholders so as to gain a plurality of perspectives regarding program impact. It also ensured that data gathered was not solely anecdotal but credible, dependable and confirmable [30]. Within this overall aim for increased creativity and rigor, the research remained subject to the standard complexities and constraints of limited time, budget, data and personnel [31] [32]. It light of the constraints it was decided by the MoEST that five representative test schools would serve as a sample of the 50 participating in the program. The schools selected by the MoEST were Chin’gombe, Mwatibu, Mthentera, Mbinzi and Dzenza. None of these schools had any form of ICT incorporated into the curriculum prior to the introduction of the program. A total of 15 days were spent in these schools due to a recognition of the benefits of prioritizing classroombased research [33]. A combination of qualitative and quantitative methods were utilized and these are now outlined alongside the rationale for each. A total of 15 lessons in which the learning machines were being used were observed throughout the process [34]. Teachers were asked to conduct the lesson as they would normally, without altering classroom arrangements. Observing usage of the technology in a normal environment provided an ideal foundation for the subsequent methods, giving opportunity to discuss successes and suggest ways to work around challenges with the children, teachers and headteachers. Some 15 group interviews were held with children in the five schools, in order to hear what difference the program had made to their lives. The children were asked what they liked and disliked and improvements they would make for the future. The interviews followed a semi-structured, guided approach [35] [36] facilitated by a teacher and conducted in Chichewa. Participants were selected at random from Standard 3 and 4 classes, were aged between 7 and 15, and had equal gender representation. Following these group interviews, more detailed conversations were held with individual children in order to learn about particular experiences or perspectives that they had expressed. Such unstructured and story-based conversation was useful in identifying unanticipated program


impacts [37]. In addition to this, a three stage individual interview was conducted with the head-teacher of each school and also with four related officials from the MoEST. These interviews provided opportunity to obtain input regarding the program feasibility, impact and future direction. Conducting focus groups was considered to be the most appropriate method to gain detailed feedback from teachers regarding their view of the program and the difference it had made to their lives, including positive and negative impacts and potential improvements. A total of 15 focus groups were conducted in this safe environment [38] which facilitated robust dialogue [39] but ensured no one felt pressurized to share personal experiences [40]. Building on these, the learning octagon was a research tool developed specifically for the program in order to stimulate more detailed discussion between teachers in the focus groups. The octagon providing a pictorial representation to assess the strengths and weaknesses of eight different dimensions to the program by drawing on a combination of the Octagon tool [41] and Most Significant Change approach [42] [43]. This enabled the teachers to collaborate on creating a visual representation of the impacts of the initiative. In addition to these methods, and in order to enable ongoing monitoring on a weekly basis, one teacher from each of the five schools was selected to complete a diary documenting their experiences of using the learning machines in their lessons. The diaries were semi-structured and designed to provide a continuous record of particular strengths, weaknesses and challenges encountered. Baseline tests were conducted at stage one and three of the research. The objective of the test was to provide a quantitative assessment of the impact of the program on the attainment of the children regarding both curriculum and lifeskills. In each of the five test schools there were 12 children randomly selected from Standard 3 and 4, providing a total of 60 children, 30 boys and 30 girls. Five additional schools were selected to act as a control group with an additional 60 children tested. Alongside focusing on the test schools, evaluation questionnaires and equipment feedback forms were distributed to all 50 participating schools in order to gain a broader understanding of program impact. The questionnaires gave opportunity for feedback regarding patterns of usage and challenges encountered in implementation. The equipment feedback forms enabled the documentation of technical problems encountered throughout the program. Each of the methods outlined above enabled a detailed perspective to be developed regarding the strengths and weaknesses of the program. The monitoring and evaluation methodology employed was participatory throughout [44] and engaged with children, teachers, headmasters, community leaders, government officials and civil society representatives. Each school was visited at least three times throughout the six month research period and this allowed for a progressive approach and the development of good relationships with a number of stakeholders.

V. ANALYSIS The monitoring and evaluation exercise was focused on assessing three interconnected spheres which cut across the five previously identified research objectives. These were impact upon students, impact upon teachers and effectiveness of the technology. Analysis of each sphere is based on the data gathered using the range of methodological approaches previously outlined. The analysis is illustrated throughout with quotations from children, teachers, head teachers and MoEST officials. A. Impact upon Students In assessing the overall impact upon students the study considered the specific impact on four main areas of student attendance, enthusiasm, attainment in curriculum subjects and attainment in life skills subjects. A significant and universally agreed impact of the program was the increase in school attendance. Teachers reported a large increase in class sizes with fewer children absent than prior to the intervention. One student from Standard 4 explained why this was the case: Before the gadgets more pupils were absenting themselves from classes but now we encourage our fellow pupils to come to school and tell them, today if you absent yourself, you will miss using the gadgets. I used to absent myself 50% of the time before the gadgets came, now I come to school everyday. This is a significant achievement within a national context of high drop-out rates and absenteeism. For any educational intervention to succeed it is necessary for learners to be attending school on a regular basis and the motivating influence of appropriate technology is clearly a contributing factor in achieving this desired end. Despite strong anecdotal evidence, assessing the statistical significance of the change in attendance was hampered by a lack of daily attendance records maintained in each school, meaning it was impossible to track exact attendance patterns prior to the current term. Children in the group interviews reported that they were enthusiastic about coming to school now that they were using the learning machines. Several students reported having shared the experience with family and community members who actively encouraged them not to miss out on the opportunity to learn with the new technology. As one student from Standard 3 explained: They [parents] said to me that you have to work hard and make sure you do not run away from lessons when you are using the computer. I used to run away from class but now I have changed my behavior … because I am attracted to the gadgets and if I miss the chance to use it, it will never come again. Other children emphasized what they had learnt through the lessons and how this related to what they hoped to do in the future. One student from Standard 5 reported the impact of a particular lesson regarding counting:


Now I know how to add and subtract – I could see the pictures and I was able to subtract. This makes a foundation for me to be able to work in a bank – this is what I want to do in the future. The learning machines are perceived by the children as a mixture between mobile phones, games devices and video players. This makes them attractive to the children, who quickly become familiar with them and enjoy the process of being congratulated by the device for answering questions correctly. Observed learners appeared to enjoy working in groups and taking turns in pressing the buttons. Teachers also noted a similar impact and emphasized the improved listening skills of the students: They are able to explain things now – and the gadgets really help them with listening skills – if they do not listen then they miss what has been said and they cannot answer the question – and it makes them be fully attentive. Increased attentiveness in class and greater motivation to attend school also had an effect on the attainment of the children in both curriculum and life skills based lessons. Both students and teachers reported that the use of audio and video, as well as the continuous assessment quizzes, increased retention and affected attainment. One teacher reflected on tests completed the previous week: We had the mid-term tests last week and more pupils did well than before – when I asked them why they said it was because of the gadgets – ‘we just remember what we have learnt on them’. The increase in attainment has gone up by about 30% on average from what they normally achieve. Despite such assertions it was difficult to assess quantitative impact on student attainment due to the short period of time since the beginning of the intervention. A comparison of scores between the baseline and second test demonstrated that certain sectors of the curriculum had considerably more correct answers after using the learning machines. However, the majority of students had only completed 15% of available lessons at the time of the second test and the overall improvement in curriculum attainment was limited to 1.5%. Similarly, improved attainment in life skills was difficult to quantify as test scores related to knowledge gained rather than necessarily to behavioral and lifestyle change. The overall impact on life skills attainment in the baseline test was a 3% improvement. Again, the questions which were answered correctly indicated an increased understanding of the content for those lessons which the teachers had chosen to use most regularly. Although quantitative change proved difficult to assess, many students interviewed were able to recall accurate information on the life skills lessons they had used and expressed satisfaction to have learnt skills useful to their daily lives. Learners explained how they had put into practice what had been taught in the lessons, indicating potential behavioral change: [my favorite lesson was…] Preventing malaria – because it affects many children in the village. I learnt that we must sleep under treated bed nets and must not play with stagnant

water. I did not know this before the gadgets. Children were also able to talk about culturally sensitive subjects such as HIV/AIDS and explain what they had learnt from the lessons. Teachers stressed the value of the interactive lessons in a cultural context where such discussions are often considered taboo. The children requested that new content be added to the devices so that they could learn about a greater variety of topics. The most commonly requested subjects were Agriculture, Mathematics, English, Religious Education, Science and Technology, Physical Education and Music. Similarly, the teachers requested that new content would remain linked with the curriculum reform in order to provide them with support in this transition. The impact of the initiative on the students is intrinsically linked to the way in which the teachers are affected, and this is now considered. B. Impact upon Teachers In assessing the overall impact upon teachers the study considered two factors, teacher enthusiasm and teacher workload. The majority of teachers were enthusiastic about the introduction of the program into their school and several commented that they felt honored to have been selected as a school for the pilot. They were pleased that MoEST officials were visiting their school and showing an interest in their work. They were keen to learn about technology and could see the positive effects on themselves and the learners. They also noted how the introduction of the learning machines had helped them in altering their teaching style, adopting new and innovative approaches: We love it – before the gadgets it was just talk and write for some subjects – but now the kids can see the pictures. Some of the children can be sitting doing one thing while others are doing the gadgets. Young trainee teachers particularly enjoyed using the technology and often engaged in using it without apprehension. This confidence had a clear effect on the manner in which the children approached the devices and made use of them in the classroom. The enthusiasm of the students also affected the teachers: …because if the learners are enthusiastic then it makes us enthusiastic. If the gadgets help the students to learn then the teachers have to be happy that the students are doing well. However, this enthusiasm was not felt universally. Mature teachers often showed less interest in the program, were fearful of using the technology and reluctant to let the children use it independently. In one of the test schools after six months of usage there were only three teachers still using the learning machines. All the other teachers in the school had become uninterested in the program, considering the additional workload to be too much of a burden. The teachers reported a variety of different experiences regarding the impact of the program on their workload. This was dependent upon the manner of implementation in each


school and the degree to which the learning machines were incorporated within the lesson schedule. The overall feeling from the teachers was that the introduction of the gadgets had increased the workload in certain areas and decreased it in others: There are two things, lesson preparation - for this it might take more effort - and lesson delivery - for this it might take less time. It has made us do more work – we have to guide the students. And there are not enough gadgets – so if we had enough gadgets then it would be ok and then it would not increase our workload. Since the intervention was a pilot project the majority of schools chose to use the gadgets after the school day had finished so as not to interfere with the daily timetable. This meant that the teachers were required to stay in school for an extra hour every day to facilitate the lessons. Most were willing to do this but some expressed reluctance due to preexisting after-school commitments. It was also commented in two schools that the initiative resulted in more preparation work for the teachers because they needed to become fully familiar with the devices themselves before making use of them in lessons with the children. In the two schools where the teachers had chosen to incorporate the gadgets into the school day then a decrease in workload was reported. In these cases the gadgets helped with effective facilitation, the management of large classes and provided support in preparing teaching and learning materials for the lessons. In light of this, the program would benefit from the development and distribution of a pedagogical guide for teachers regarding the effective integration of the learning machines into the learning routine of the children. Teachers should also be encouraged to utilize the full range of lessons that are available on the devices. This would complement the development of a schedule for teachers to use the machines during school time, giving clear explanation how it can most appropriately support the curriculum. The increased ability of the children to memorize what had been taught was also recognized to make the job easier by one teacher from Standard 4: Gadgets have simplified the work for the teachers – when the child sees something he remembers – when he just hears he forgets. When they see it, it stays in their mind. Others noted that the impact of the learning machines in encouraging so many children to return to school had caused class sizes to expand significantly, resulting in more work attempting to manage the children. A common concluding comment from teachers was that the workload would decrease significantly if there was a lower pupil to gadget ratio and if the lessons on the gadgets were fully compatible with the syllabus and incorporated into the curriculum. The program would therefore benefit from an increased allocation of 20 learning machines per school. This would facilitate a more concentrated impact and allow a greater number of children in smaller groups to benefit from the content on the devices. In light of the challenges presented, it is clear that regular training sessions for the teachers would assist in maintaining

the momentum of the program and improving teacher motivation. The limited budget of the MoEST makes it a constant challenge to deliver adequate training, especially in remote areas. The teachers requested opportunity to ask questions and provide feedback, requiring effective communication channels between the stakeholders. As highlighted by one official: If you come back frequently then the schools will know you are coming and they will keep using the gadgets but if they think no one is coming to visit them then they will forget to use them. This is because when they are monitored they will feel that somebody is appreciating their job – they are happy that they can be involved and they see that somebody cares. The feedback from teachers demonstrated that sustained monitoring from MoEST officials constituted a significant capacity building process. The impact was especially pronounced in marginalized rural areas which were inaccessible and rarely visited. In light of this is it clear that a continuing focus on teacher training which both develops skills and instills value is intrinsic to the success of this and any ICT for education initiative. The ability of the teachers to utilize technology with confidence and adapt it to the specific needs of the classroom is a central determining factor. Without this integration any initiative remains an appealing add-on but does not have significant effect on the culture of the classroom, pedagogy, or rationale for learning. C. Effectiveness of the Technology The success of the initiative, whilst not determined by the technology, was dependent upon it operating effectively within each school. Usability, durability and the reliability of the devices and charging solution were the three areas assessed in the study. The lesson observations demonstrated that the majority of children were able to operate the learning machines independently, understanding the purpose of the buttons and completing quizzes without supervision. The process of lesson selection occasionally required guidance from teachers but this was rarely necessary because the most capable children were seen to take a lead in the learning groups, demonstrating considerable confidence with the devices. However, a significant factor limiting usability was the low number of devices per classroom and subsequent high number of students per group, on occasion as many as ten learners. This was not conducive to learning as the machine is designed for small groups, with small screen and quiet audio output. In such situations, dominant children were observed bringing the machine to their ear in order to hear the lesson, meaning other children were unable to engage and became easily distracted. This was confirmed by a participating official from the MoEST who noted: Bringing such technology is very important - it is only that at the moment the gadgets are very few – there needs to be more. It is important that you have the right number of gadgets for the number of children because it is difficult for so


many children to share. Due to the nature of such a pilot program it was anticipated that there would be significant problems regarding device durability. However, after six months of usage only 8% had developed problems, the majority being due to software issues easily resolved through reformatting. Two of the devices stopped working due to the on/off switch breaking, these were repaired in situ and left fully functioning. A major concern in this respect was that teachers often assumed they had broken the devices when all that was actually required was for them to be charged fully. The solar panels operated effectively in each of the five test schools throughout the program with all of them maintaining adequate power to charge the devices as often as necessary. Several program schools outside the five included in the test did report problems including two faulty batteries, one loose connection and one stolen panel. In addition to these, one school adopted the practice of disconnecting the battery from the panel to remove it for safe keeping during the night and this prevented the battery from charging fully. VI.


The analysis of impact on students, teachers and technological effectiveness has demonstrated the educational potential of the Interactive Learning Program. The introduction of the devices had significant positive impact upon school attendance and levels of enthusiasm. Also noteworthy was the increased value placed on teachers and the role of education within the community. However, alongside this, considerable challenges were encountered including the negative impact on teacher workload, lack of classroom integration, significant infrastructural constraints and the necessity for further teacher training. The future success, sustainability and potential for project scalability is dependent upon engaging with each of these identified challenges. The observations made in the analysis speak directly to the program efficacy but are also applicable lessons for ICT for education initiatives more widely. The introduction of technology into education systems in the developing world is a complex procedure with considerable potential for failure and therefore it is vital that rigorous monitoring and evaluation is incorporated throughout all such initiatives. The price of portable educational technology is constantly decreasing, making initiatives such as the Malawi Interactive Learning Program increasingly cost effective. The September 2008 price of each learning machine was circa US$55 and as each device can be used by several children the per child cost is considerably lower than other widely publicized solutions. Significant future potential also lies in the opportunity for integrating the educational content with new generation mobile phones and other handheld devices becoming progressively more available across the region. Linked to this, increased ability to send and receive data in standard formats (for example Macromedia Flash, xml, mp3 or mp4) will allow

for the development of user interaction. Market influences on the development of mobile technology also ensure sustained improvements in hardware capabilities and the subsequent emergence of new educational possibilities. However, within all such initiatives, it is necessary that the application of technology is conceptualized as a tool in facilitating the overall aim of catalyzing a more fully effective approach to education. In reaching this goal many of the challenges documented remain the same regardless of the presence of technology. It is not overly surprising that the introduction of ICT into primary schools in Malawi caused a dramatic increase in attendance figures. The significant question for ongoing research remains one of long term impact on approach to schooling, once initial enthusiasm surrounding the program has subsided. Whilst a significant achievement, increasing school attendance figures is not the most significant long-term challenge facing education in Africa [5]. Instead, it should constitute an initial building block which can be capitalized on through adopting an integrated approach to developing the capacity of the education system. One dimension to this will be the effective utilization of curriculum enhancing technology, such as the Interactive Learning Program. For full benefits to be realized, such programs require transition from their position of peripheral curiosity to one of being integrated as a sustainable teaching tool. Future research is required to assess the degree to which this is occurring in Malawi. A variety of different solutions are required to address the challenge of providing equitable access to good quality education in the developing world. The increasing availability and decreasing cost of portable devices ensures that they are likely to play a prominent role in the future, however this should not lead to such technologies being presented as an educational panacea. For programs to be of maximum educational benefit in Malawi, as in many such initiatives, the critical issues for consideration and action remain pedagogy, classroom integration and teacher training. REFERENCES
[1] [2] [3] [4] [5] [6] MoEST, Link for Education Governance. Education Sector Performance Analysis. Lilongwe, Malawi. 2007. World Bank, Data and statistics: Malawi. 2006. www.web.worldbank.org Accessed 21/08/08 MoEST, Primary Curriculum and Assessment Reform (PCAR). Implementation Plan 2007-2010, Lilongwe, Malawi. 2008. UNESCO, Institute for Statistics, country data 2008. www.uis.unesco.org Accessed 21/08/08 EFA, Global Monitoring Report: The quality imperative. 2005. www.portal.unesco.org/education Accessed 21/08/08 D. Pye and J. Stephenson, Using ICT to Increase the Effectiveness of Community-based, Non-formal Education for Rural People in SubSaharan Africa: the CERP Project Final Report (DFID Educational Paper 50). London: Department for International Development 2003. V. Tinio, ICT in Education. E-Primers for UNDP, New York 2003. J. Leach, DEEP IMPACT: an investigation of the use of information and communication technologies for teacher education in the global south. Researching the Issues, 58. DFID 2005. D. Wagner, Monitoring and evaluation of ICT for Education: An Introduction. Chapter 1 in: D. Wagner et. al. 2005. Monitoring and

[7] [8] [9]

Evaluation of ICT in Education Projects – A Handbook for Developing Countries, InfoDev 1-19, 2005. D. Keats, The genesis and emergence of Education 3.0 in higher education: the potential for Africa. First Monday 12, 3, 2007. InfoDev, Knowledge Maps: ICTs in Education. World Bank 2005. C. Fadel and C. Lemke, Technology in Schools: What the research says. Metiri Group, Commissioned by Cisco Systems 2006. L. Cuban, Oversold and Underused – Computers in the Classroom. Harvard: USA 2001. T. James and J. Miller, Developing a Monitoring and Evaluation Plan for ICT in Education. Chapter 4 in: D. Wagner et. al. 2005. Monitoring and Evaluation of ICT in Education Projects – A Handbook for Developing Countries, InfoDev. 57-77 2005. D. Wagner, B. Day, J. S. Sun, Recommendations for a pro-poor ICT4D non-formal education policy. Final Report for, Imfundo: Partnership for IT in Education, DFID, 2004. M. Sharples, Disruptive Devices: personal technologies and education. Educational Technology Research Paper Series 11, The University of Birmingham, 2000. H. U. Hoppe, R. Joiner, M. Milrad and M. Sharples, Wireless and Mobile Technologies in Education. Journal of Computer Assisted Learning, Guest editorial, 19, 3, 255-259, 2003. F. Lehner and H. Nosekabel, The Role of Mobile Devices In E-Learning -First Experiences With A Wireless E-Learning Environment. WMTE 103-106, 2002. J. Leach, T. Power, R. Thomas, X. Fadani and A. Mbebe, 4D Technologies: appropriating handheld computers to serve the needs of teachers and learners in rural African settings. Centre for Research and Development in Teacher Education, Open University: UK, 2003. C. Pontefract, Learning to Share, Insights Education 1 Missing the Connection – using ICTs in Education, February 2003 R. Dhanarajan, Learning Technologies, where is the challenge? Education, Communication, Information 1, 1 Spring 2002 MoEST, Primary Curriculum and Assessment Reform (PCAR). Literature Review of Primary Education in Malawi, Lilongwe, Malawi, 2002. G. Bunyi, Rethinking the place of African indigenous languages in African education. International Journal of Educational Development 19, 4-5, 337-350, 1999. EFA, Global monitoring report: Literacy for Life, 2006. www.portal.unesco.org/education Accessed 21/08/08 R. Kozma and D. Wagner, Core Indicators for Monitoring and Evaluation Studies for ICT in Education. Chapter 3 in: D. Wagner et al. 2005. Monitoring and Evaluation of ICT in Education Projects – A Handbook for Developing Countries, InfoDev. 35-57, 2005. J. Taylor and S. Soal, Measurement in Development Practice: From the Mundane to the Transformational. Chapter 5 in L. Earle (ed) 2004. Creativity and Constraint: Grassroots Monitoring and Evaluation and the International Aid Arena. INTRAC Policy Series 18, 95-110, 2004. P. Morgan, Measuring the Development of Capacity: Is this Still a Good Idea? Chapter 2 in L. Earle (ed) 2004. ‘Creativity and Constraint: Grassroots Monitoring and Evaluation and the International Aid Arena’. INTRAC Policy Series 18, 47-62, 2004. D. Watson, Monitoring and evaluation of capacity and capacity development. ECDPM, Discussion paper 58B. April 2006 D. Cabrera, A Theory of Systems Evaluation. Systems evaluation and evaluation systems whitepaper series. New York: Ithaca, Cornell University, 2006. J. Baxter and J. Eyles, Evaluating qualitative research in social geography: establishing ‘rigour’ in interview analysis. Transactions of the Institute of British Geographers, 22, 4, 505-525, 1997. M. Bamberger, J. Rugh and L. Mabry, Real World Evaluation. Working Under Budget, Time, Data and Political Constraints. London: Sage, 2006. D. Gray, Doing Research in the Real World. California: Sage 2004. M. C. O’Sullivan, What is happening in the classroom? A commonsense approach to improving the quality of primary education in developing countries. Teacher Development, 9, 3, 301-314, 2005. J. W. Creswell, Educational Research: planning, conducting, and evaluating quantitative and qualitative research. New Jersey: Pearson Education, 2005. [35] J. Burgess, The Art of Interviewing. In: A. Roger and H. A. Viles, (eds) The Student Companion to Geography. Blackwell: Oxford, 242-249, 2003. [36] M. Q. Patton, Qualitative Research and Evaluation Methods. California: Sage, 2002. [37] R. Chambers, Participatory Workshops – a sourcebook of 21 sets of ideas and activities. United Kingdom: Earthscan, 2002. [38] G. Valentine, Being seen and heard? The ethical complexities of working with children and young people at home and at school. Ethics, Place and Environment. 2, 141-155, 1999. [39] D. Stewart and P. Shamdasani, Focus Groups: Theory and Practice. Newbury Park: Sage, 1990. [40] P, Bhattacharjee, Stepping Stones – A participatory tool to integrate gender into HIV/AIDS work. Development in Practice, 10, 5, 691-694, 2000. [41] Sida. The Octagon. A tool for the assessment of strengths and weaknesses in NGOs, 2002. [42] P. Sigsgaard, Monitoring without indicators: an ongoing testing of the MSC approach. Evaluation Journal of Australasia. 2, 1, 2002. [43] J. Dart & R. J. Davies, A dialogical story-based evaluation tool: the most significant change technique, American Journal of Evaluation. 23, 2, 137-155, 2003. [44] M. Huberman, The Many Modes of Participatory Evaluation. Chapter 7 in J.B. Cousins and L. M. Earl (eds) 1995. Participatory Evaluation in Education. Studies in Evaluation use and organisational learning. London: Routledge, 103-114, 1995.

[10] [11] [12] [13] [14]

[15] [16] [17] [18] [19]

[20] [21] [22] [23] [24] [25]



[28] [29] [30] [31] [32] [33] [34]


Claim Mobile: Engaging Conflicting Stakeholder Requirements in Healthcare in Uganda
Melissa R. Ho, Emmanuel K. Owusu, and Paul M. Aoki
fraud and transformed supply-chain management for the Echoupal project [6]. While health information is critical to the improvement of healthcare in developing regions, financing healthcare also remains a significant unsolved problem. Can we take lessons from e-Choupal and apply them in the healthcare sector? The design of usable, reliable, and fraudresistant tools for management of these aid flows is an area with potential for very significant impact. However, in the case of healthcare, the financial models are very different from commercial markets – financing of healthcare typically comes through transnational aid agencies like the World Bank and International Monetary Fund (IMF), and is implemented by non-governmental organizations (NGOs) and the local government. Since the NGOs are typically experts in health, not technology, data processing is often outsourced to third-party information technology (IT) vendors. Relationships between the vendors, the NGOs, the local governments, and the transnational aid agencies are not always smooth - and limitations in communications infrastructure means that the information flows between them are scattered at best. In this paper we suggest that the “closed loop model” generally used by researchers in deployments of mobile health applications does not map onto the financial and political realities of the mainstream of healthcare provision in Africa, and limits the ability of pilot programs to increase their scale and impact. We describe an innovative, IT-based, NGO-run healthcare access program in Uganda, and our experiences designing and deploying Claim Mobile, a mobile-phone based system intended to address inefficiencies and help the program scale to additional districts. We argue that in addition to addressing the needs of the primary users in the system, the health workers, our design must consider the requirements, motivations and concerns of the other stakeholders: the IT vendors, the NGOs, the government, and the aid agencies. Our designs must consider the larger order ramifications of how we may positively and negatively impact both the “users” who will be generating the data, and the entities that will be engaged in managing and using the information in the resulting database. Just as the e-Choupal project assimilated the middlemen by hiring them as kiosk operators, we propose that we can design applications structured to accommodate conflicting stakeholder requirements, while also alleviating information inequalities resulting from limitations in the system prior to the introduction of the information technology.
Abstract—Claim Mobile is a platform designed to support a project that subsidizes healthcare by reimbursing health service providers in Uganda for treatment of patients with sexually transmitted infections. As with many development projects, the Uganda Output-Based Aid (OBA) project involves a number of stakeholders: the service providers, the project implementers, the financiers, and the Ugandan government. Design of an appropriate solution requires meeting the various and conflicting requirements of all of these stakeholders. In this paper we detail the rapid design and testing of a pilot implementation of a mobile and web-based system for processing claims forms, based on two prior field visits to Uganda. Based on a comparative device study, semi-structured interviews, health clinic surveys, and a brief deployment, we affirm the selection of the mobile phone as a platform from the health clinic perspective, and further suggest that effective design for development requires more than addressing requirements of the the “users” of the mobile phones but also all the other stakeholders involved, who may have conflicting requirements. Index Terms—mobile phone, ICTD, health, participatory design, Africa, HCI

I. I NTRODUCTION Mobile phones are frequently touted as being the appropriate and sustainable platform for rural healthcare in Africa. They are relatively cheap, durable, consume less power than laptops and desktops, and incorporate a battery that makes them more amenable to use in places with intermittent or no power. Commonly proposed uses are for data collection [1], [2] and decision support for rural health workers [3], [4]. Some projects also use mobile devices as a platform for information dissemation as well as data gathering [5]. However, these are all generally “closed loop” systems in which researchers are able to control all aspects of the system design and operation, focusing their research primarily on the rural health workers that will be using the mobile phones. Other applications have even more potential for large-scale impact. In the agricultural sector, we have observed how the introduction of transparent market prices and subsequent hiring of “middlemen” to purchase from farmers has reduced
Manuscript received October 1, 2009. This work was supported by the Blum Center for Developing Economies and the U.S. National Science Foundation Summer Undergraduate Program in Engineering Research at Berkeley (SUPERB) under Grant No. 0453604. Melissa R. Ho is with the School of Information at the University of California, Berkeley, 94720, USA. (phone: +256 777 723 786; email: mho@ischool.berkeley.edu). Emmanuel K. Owusu is with the Computer Engineering Department at Iowa State University, 50011, USA. (email: kwame@iastate.edu). Paul M. Aoki is with Intel Research, Berkeley, CA, 94704, USA (email: aoki@acm.org)


Mbarara, Uganda • HIV prevalence: 10% of adult population (15-49 years) • Syphilis prevalence: about 5-7% of adult population • 1 in 4 households had at least one phone. • 39% reported STI symptoms • only 1/3 sought care • 54% of respondents who sought any STI treatment reported using private clinics.
Fig. 1. A HealthyLife voucher. The ‘M’ in the top left is a note written by the health service provider to indicate that the first client voucher on the left was given to a male client, and that therefore the partner vouchers on the right should be given to a female client. TABLE I S OME BACKGROUND STI STATISTICS ABOUT M BARARA , U GANDA [7].

II. BACKGROUND Providing effective health care in poor countries is an essential component to economic development and poverty reduction. Unfortunately donors supporting this endeavor often find that resources given are not matched by desired gains. The output-based aid (OBA) model of financing seeks to address this by paying healthcare providers directly for services rendered instead of paying for the service provision up front. However, OBA program management is information intensive, necessitating much paperwork to track and reimburse payment claims. Smartphones (mobile phones with advanced features such as the ability to run third-party software) have the potential to alleviate this burden. In collaboration with a local NGO and their partnering IT vendor, we have proposed to deploy a number of smartphones for use in an OBA project based in Western Uganda, with dual goals of reducing claim processing time and improving communication between the health care providers and the OBA management agency. The project is managed by the local branch of a multinational NGO and a for-profit health insurance company, in collaboration with the Ugandan Ministry of Health (MoH) and Ministry of Finance (MoF). The project is primarily funded by an aid agency based in Europe, with additional funding for the expansion coming from a separate transnational funding agency. Together, they run a voucher program called HealthyLife, which treats sexually transmitted infections (STIs), reimbursing providers for the diagnosis and full course of treatment only after the patient is seen. This program was implemented in response to the high burden of sexually transmitted infections in Uganda, and began in July 2006 in four districts of southwestern Uganda: Mbarara, Ibanda, Kirihura and Isingiro (See Table I). Patients buy treatment vouchers in pairs, one for the client and a second one for the client’s sexual partner (See Figure 1). Each voucher is good for one consultation (generally including a lab test to diagnose the STI) and three follow-up visits. During the consultation, the provider completes a claim form recording the client’s demographics, the examination and laboratory results, a diagnosis and details of the course of treatment prescribed (See Figure 5). Completed claims forms are sent to the voucher management office in the city of Mbarara, the main urban center of Western Uganda. Forms

can take two weeks or more to move from the providers office to the management agency. The current data management system requires all claims to be submitted on paper forms to the management agency. At least another two to four weeks are spent reviewing each claim, cleaning data from improperly-completed forms, and verifying that the service took place among suspect claims. Two months or more can go by before the provider is reimbursed for service provision. In Uganda, private providers traditionally operate on a feefor-service model, receive prompt payment, and do not have a large operating margin. In many cases, payment is provided prior to service. Delays in payment result in delays in procurement of replacement prescriptions and medical supplies, often leading to a temporary hiatus in service. Encouraging provider involvement in the OBA program requires a great deal of confidence on the part of the providers to participate. If a system to shorten claims processing could be devised, more providers could join the scheme and more patients could be provided the life-saving STI treatment voucher subsidy. The remainder of this paper details the system we are currently piloting, in which claims are submitted via Internet from a mobile phone directly to all the parties in the management agency. In addition to describing our user studies and how this has informed the design of the system, we discuss the problem of negotiating conflicting stakeholder requirements. We find that in projects with multiple stakeholders, the introduction of a system may disrupt balances of power, particularly around the flow of information and money. As a result, the design of this system, in order to secure positive support from all parties involved, must carefully balance stakeholder incentives. III. M ETHODS The research described here involved an iterative process of field research and prototyping. The fieldwork and deployments have been done over the course of three visits to Uganda: an initial two-week visit in Summer 2007 to establish a relationship with the project, in which we also conducted a survey of the clinics in the program; a followup visit for three weeks in November 2007; and a five-week pre-pilot deployment in August-September 2008. During all three visits we conducted semi-structured interviews with the various stakeholders, and directly observed claims form entry and processing. When given permission, we did audio and video recording of interviews and user study activities. In all, we


have approximately 30 hours of audio, and have done detailed interviews in seven of the 12 participating clinics (in addition to the initial survey of all of the clinics), as well as intensive observation in two clinics, a rural, high-claim-volume clinic with very little exposure to computers, and an urban lowclaim-volume clinic with its own computers. The last visit entailed a comparative user study as well as deployment of the mobile phones in the latter two clinics. A. Clinic Surveys The clinic survey was conducted in conjunction with a larger survey of available infrastructure at contracted clinics. We asked 14 questions, assessing familiarity with computers and mobile phones, but primarily gathering feedback from the health clinics on the claims process (e.g., how long it takes them to fill out the paper forms, and what their priorities might be for improvement of the process). We also collected various documents from the management agencies regarding the performance of each clinic, including all available financial reports on processed claims, and in November, we returned to seven of the clinics to do in-depth surveys and to follow-up on the survey findings. B. Rapid Development and Pre-pilot Deployment Initial prototyping occurred in early 2008, and we returned to Uganda in Summer 2008 to do a three-stage pilot deployment, first testing the functionality of our software, second reviewing the proposed claims process with the management agencies, and finally taking the phones to the health clinics to test the mobile phone interface in the field. During this time we also conducted another round of semi-structured interviews to gather information on changes in the claims submission process (for example, claims processing had moved from Mbarara to the national capital, Kampala). We did iterative development based on feedback from the various stakeholders, trying out features as they were suggested, and developing new tools as seemed merited by findings in our interviews. To gain a more in-depth understanding of health clinic life, we stayed overnight for three days in the rural health clinic, thereby supplementing the the semi-structured interviews with direct observation of actual practice. The primary purpose of this last field visit was to conduct a pre-pilot demonstration, using the mobile phones to submit actual claim data to the management agency, have it reviewed, and have the management agency provide feedback to the health clinics via the mobile phones. We simulated the proposed process, physically following the claims forms from the time the patient comes into the health clinic, through the preparation of the claims summary forms, physically transporting the forms to the management agency where we observed the claims approval, and data entry into the existing database. We simultaneously had the service providers submit the claims form via Claim Mobile, enabling the management agency to provide feedback to the service providers through the system. The pre-pilot is still operational, with mobile phones remaining in the two clinics, and the full pilot will
Fig. 2. This diagram illustrates the flow of money and information between selected stakeholders in the OBA project, both currently, and how it will be once the Claim Mobile system is fully deployed.

be conducted in May 2009, with mobile phone-based claims processing expanded to 8-10 additional clinics. IV. S TAKEHOLDERS In this section we describe the funding, management and service provider organizations to which we alluded in Section II. Fundamentally, all stakeholders want to improve STI treatment and reduce the prevalence of STIs. Each stakeholder also has a financial interest in the success of the overall project - staying afloat for the health service providers, and staying within the aid agencies’ target budgets for the management agency partners. The discussion highlights the ways in which the various stakeholders have competing as well as common interests. While we will detail several stakeholders in this section, there are six key stakeholders: the aid agencies who fund the OBA project, the financial management agency (FMA) which receives the funds from the government and disburses them, their program management office (PMO) in Mbarara which runs the program and interacts directly with the health service providers, the technical management agency (TMA) that manages the claims processing, the health service providers (HSPs), and us, the mobile platform developers (MPDs). Figure 2 illustrates some of the relationships between these entities which we will describe in detail in the remainder of this section, based on qualitative fieldwork and document analysis. A. Aid Agencies As the funder of the HealthyLife STI treatment program, the involvement of the European aid agency is more than apparent. Their role in the management of the program is more supervisory – a consultant goes to Uganda at irregular intervals to help with planning of the program, and they do some monitoring. They also have commissioned another nonprofit, affiliated with a North American university, to conduct an evaluation of the program. Ultimately, however, they control the flow of money to the financial management agency, which then pays the IT vendor to handle the technical aspects of the operation. In the past year, the European aid agency has worked with an additional transnational aid agency to fund the expansion


of the project into additional districts. While they may not have a direct impact on the information processes in the project itself, the funders’ internal actions have direct impact on the project as a whole. In one example, a delay in payment to the European aid agency resulted in a delay in payment to the two management agencies. As a result the IT vendor ceased processing of claim forms until payments were received. However, not only did the voucher program grind to a halt, but reimbursement to the participating providers for patients already seen was delayed as well; the management agencies ended up with a backlog of claims forms to process, exacerbating the length of time it takes to process claims and further delaying payment for services. B. Financial Management Agency (FMA) The local NGO partner that acts as the financial management agency (FMA) is the Uganda office of a multinational non-profit sexual and reproductive health organization with a goal of reducing unintended pregnancies and unwanted births through family planning and other methods. Their role in this project is to receive the funds from the aid agencies via the Ugandan government, using these funds to pay the health clinics and to pay for other program costs, including the database software development and management. The main office in Kampala runs this program (as well as others) and manages several clinics throughout Uganda, one of which was a participating clinic in the HealthyLife program until Summer 2008. In addition, they have a program management office (PMO) in Mbarara which is directly in charge of coordinating the the OBA project. In the claims process, the FMA disburses payments to each of the service providers, based on claim reports from the technical management agency (TMA). C. Program Management Office (PMO) The HealthyLife PMO in Mbarara has five full-time staff, in addition to two people that help with cleaning and cooking, and the FMA staff that come in and out of Kampala for related programs. There are two computers in the office, one in the project coordinator’s office, and another in the finance office, shared by the Behavior Change Campaign (BCC) coordinators who go out into the field to run community radio advertising programs and to distribute vouchers. Their Internet connection was down when we arrived, but was repaired the same day and largely functional for the remainder of our four weeks there. They share a 56kbps dial-up connection over a local area network. It became clear through our interviews in this office that, while the PMO is the nominal clearinghouse for information between the TMA and the health clinics and is primarily responsible for communication with the health clinics, they actually have the least information of all of the stakeholders in the OBA program. At the point in which the database processing moved from Mbarara to Kampala, all of the claims information moved there as well. They have been able to change the claims process such that the health clinics submit two copies of each claim to the management agencies, one

for the PMO, and one for the TMA. However, the copy that remains in the PMO does not have the voucher number, a critical piece of information, and with stacks of hundreds of claims per month, the information is not in a format actually accessible to the program office until the TMA sends back claims summaries. However, even this is stripped-down and only includes the value of the claims, without any patient or diagnostic information from the claims. This poses a problem for the PMO staff’s interaction with the health clinics. They lack sufficient information to effectively counsel and train the clinics, and often feel like they do not know what is going on with the program because they do not have access to the claim data for the long claims processing cycles. In the version of Claim Mobile developed in early 2008, we intended to make the claims process more efficient by enabling the mobile phones to submit claims directly to the database (originally co-located in the PMO, now located in Kampala). In the Summer 2008 interviews it became clear that having the claims data bypass the PMO staff would deny them even more of the information they need in their interactions with the clinics. Based on this realization, we discussed the possibility of an intermediary application, a website in which the project coordinator in the PMO would be able to view claims as they are submitted, as well as any status updates. Furthermore, Claim Mobile could facilitate another of the project coordinator’s key roles in the OBA ecosystem: as the primary interface between the health service providers, the project coordinator would also be able to send messages to the service providers through Claim Mobile, either as individual messages, or broadcast announcements. D. Technical Management Agency (TMA) The TMA is a for-profit health insurance agency based in Kampala, providing conventional employer-based health insurance for the formal private sector as well as conducting nonprofit health management for targeted low-income informal sector populations. That is, their work for the HealthyLife program is in addition to their private health insurance program, and is part of a company effort to help deliver quality affordable healthcare to the poor. Their business is highly technical, and they have a wholly owned software company based in Chennai, India. As the IT vendor, the TMA’s responsibility in the program is to provide the claims forms, and the Voucher Management Unit System (VMUS), the database implemented by their software company to cross-check the claims and to generate reports. The TMA also prints (through another agency) the glossy color vouchers that the patients purchase in exchange for subsidized care. Although the data entry clerks and the database engineer were initially located in the FMA’s PMO in Mbarara, they are actually employees of the TMA, and moved when the TMA shifted claims operations to their offices in Kampala in March 2008. They carefully enter each claim into the database, later updating its status with information from the clinical officer


Fig. 3. This is a sample summary sheet prepared for one health clinic, showing a partially paid claim (QC11=Wrong consultation fees), and another claim quarantined for having the wrong voucher. While these summary reports can be informative, most clinics are not familiar with the quarantine codes, and they often don’t reach the health clinic for several months after the original claim has been submitted, often too late for the clinic to rectify any errors indicated on the report.

(a doctor) who “vets” the paper claims1 . They then produce two reports: a summary report for all clinics, and an itemized report (see Figure 3) for each clinic detailing the status of each claim, as well as any quarantine codes (Table II) or rejection reasons for any partially paid or rejected claims. In addition, when required, the TMA produces reports (based on the information in the database) for the FMA, the European aid agency, and the aid agency’s evaluating partners. Although these reports were not part of the original specified mandates for the operation, they have proved necessary for the program’s external evaluation, and there has been much friction over the work involved in the creation of reports. The relationship between the TMA and the FMA in this program is highly contentious. While initially they were equal partners in the program, both reporting directly to the European aid agency, changes in funding have led to a situation in which the TMA reports to and is paid by the FMA. On top of this, the funding for the expansion of the program has been delayed a number of times, from October 2007 to April 2008, and again to September 2008. While the TMA has received some payment, both the TMA and FMA have been operating without pay (but with promise of pay) since April 2008, just to keep the program running while the aid agencies work out the details of the new grant and the expanded program. This is part of the reality of dealing with aid-funded projects – unexpected delays in funding are common, and projects are subject to the vagaries of arbitrary rebudgeting. While the FMA is often powerless to address the issue, in this case, the TMA often chooses to respond by cutting off program access to the database, ceasing claims processing and refusing requests for reports, until their problems have been resolved. Perhaps in response to these database shutdowns, but officially as part of the aid agency’s project policy and the Ugandan government’s policy on software developed for government-funded projects, there is an expectation that the TMA’s VMUS software should be turned over to the project. However, since the TMA outsources development of this software to its partner company in India, this IT vendor considers its software to be part of its key assets, and sees its role in the project as a software licensor and service provider, not a
clinical officer is employed by the FMA, and was terminated in March 2008 due to temporary lack of funds.
1 The

software vendor. Again, while this situation is being resolved, the TMA asserts its control over the project by processing the claims, but refusing to pass on the summary reports to the FMA. While the project continues running, and the service providers continue to see patients, this introduces additional delays into the claims process, and frustrates the health clinics, whose payments are delayed without explanation. E. Service Providers: Health Clinics/Hospitals Service Providers are selected on the basis of a number of factors (e.g., services offered, capacity, personnel, geographical location). In one respect, they are the origin of the primary information in the claims management process, producing the claims records, which are then used to determine reimbursement. At the same time, as is perhaps typical, they are information-poor, because they are not given tools to use this information effectively. At the point of claims submission, they are no longer agents in the process, and must wait passively for both payments and any feedback reports produced from the information in their claims.
Code QC01 QC02 QC03 QC04 QC05 QC06 QC07 QC08 QC09 QC10 QC11 QC12 QC13 QC14 QC15 QC16 QC17 QC18 QC19 QC20 QC21 QC22 QC23 QC24 QC25 QC26 QC27 QC28 Description No indication of date of treatment No indication of time of treatment Wrong visit type: Consultation or follow-up, etc Wrong demographic information: no age and name of client Wrong/No syndrome, no diagnosis Wrong Clinical examination / not applicable to OBA Wrong/Poor diagnosis Wrong investigation/poor lab reporting Wrong drugs prescribed/invalid treatment Over prescription: more than enough Wrong consultation fees Wrong patient status: cured or not cured Next date of visit: wrong or not filled in Wrong voucher attachment/interchanged vouchers on followup visits Unclear claim/uncharged claim and treatment contradicts other visits Partner treated on client form Exceeded VMUS ceiling limit Treated syndromically and asymptomatically Unclear/wornout/blank attached vouchers Claim without patient thumbprint No voucher attachment No doctor’s signature Diagnosis contradicts clinical examination Used drugs not on OBA list Undercharged/overcharged drugs, double lab charged Patient free/normal from STI or cured not allowed for next visit Diagnosis not catered for by project Follow-up contradicts previous visits


Providers are expected to follow a rigorous course of diagnosis and treatment — they must select a lab test based on symptoms presented, and prescribe particular medications


Fig. 4. “All above denied b’cos rest of P[atien]t mgt n[ot]. ethical”: this is a sample medical advisor review of a claims summary, occurring often well after the original mistake has been made several times, before it could be caught and corrective measures could be made, as noted in the first line: “Cipro pricing b4 C[ontinuing] M[edical] E[ducation]”

on the basis of the results of the test. Any deviations from this treatment are penalized; the service providers are not paid for medications given that are not prescribed by the program. This is reasonable by public health and insurance standards, and necessary for the cost-effectiveness of the program. However, despite educational illustrative posters, training sessions, and on-site continuing education provided by the PMO’s clinical officer and project coordinator, the learning curve on the exact protocol to be used is high, and the subsequent errors are costly for the service provider. Figure 4 illustrates a particular problem in which costly errors are caught well after their first occurrence, often after the provider’s staff has have made the same mistake for a month. A few months into the program, there had been so many quarantined claims (claims that had been held for review due to discrepancies from the treatment protocol) that the management agencies and the service providers were required to do a financial settlement, in which the service providers were paid some percentage of the value of the disputed claims. Subsequently, the approval process was modified such that deviations from the protocol were partially paid (e.g., minus the cost of the incorrectly prescribed drug), and could be disputed in later reviews with the PMO’s clinical officer. Here are some figures on the value of the disputed claims for one of the rural service providers for a randomly selected month of the program:
Claims submitted: 294 Approved: 259 (88%), $1379 Approved, but adjusted: 27 (9%), $149 Quarantined: none Rejected: 8 (3%), $51 Total Requested: $1642 Total Paid: $1526 (difference: $114)

many claims are approved with incomplete data (e.g., missing demographic information for the patient). In any event, the claim form and its contents are the object of much dispute between the service providers and the management agency. Often, the service providers (especially more distant providers) feel disempowered to address this dispute and choose to accept the given variance in payment as the cost of participation in the program. The health clinic survey we conducted in July 2007 explains some of their extreme disengagement from the OBA program. Out of the twelve clinics surveyed, three said they had not gotten feedback from the OBA program at the time of the survey (July 2007), and 4 said they did not know how many claims had been rejected. While they were receiving payments on a regular basis (albeit late - 9/12 clinics defined “timely processing” as less than 15 days, half of the current processing time of 30 days), there was no mapping between the claims they submitted and the payments they received.
I don’t know. I don’t know how we are performing. I don’t know how we are faring. And of course my staffs are also complaining. They are overworked, they dont get any benefit from the project, and of course it takes a lot of time. They need to be motivated as individuals. All that will depend on – are we making any profits?

This has deeper ramifications than inability to follow up on quarantined, partially paid, and rejected claims. This communications gap between the health clinics and the OBA program leads to continued errors in adherence to treatment protocol, a feeling of lack of control over health clinic finances, and discouragement on behalf of the participating service providers. At the time of the second field visit in November 2007, we were able to follow up with the PMO and the health clinics, and noted that this situation had improved. The then-informal practice of passing on copies of the itemized clinics reports to the service providers was formalized, and clinics are receiving more feedback on their claims. However, there are physical limits to a paper and in-person based communication system, and it remains to be seen how this practice will scale as more clinics join the program. It should also be noted that the service providers differ greatly, in number of clients, setting, and availability of resources. While some clinics have computers and use them regularly, in one clinic, our smartphones were greeted with enthusiasm because they were the “first computers we have seen.” F. Patients Patients are the real target beneficiaries of the HealthyLife program. They purchase vouchers from distributors (at a heavily-subsidized price) and then go to the service provider for diagnosis and treatment. After an initial consultation with the health worker, they are directed to the lab technician, who performs the requisite test and sends them back to the health worker with a slip of paper indicating the lab result. The health worker then fills out the remainder of the form, writes a prescription, affixes the appropriate stub from the

Thus the claims submission process, while nominally a simple information flow between cooperating organizations, becomes a site of financial contention. Claims are disapproved for a number of reasons (see Table II), which could be disputed but in practice are not. However, due to the change in policy,


voucher, and has the patient sign and fingerprint the form, at which point their participation in the claims form process is complete, until they return for a follow-up consultation. For the follow-up, the service provider checks recovery progress and prescribes additional medication if necessary. In some clinics, patients are given a copy of the claim form, which they are directed to keep and bring back for the follow-up. However, most clinics do not depend on the patient copy of the claim form, and just go back through their time-ordered record book, finding the prior consultation manually. Sometimes patients either accidentally swap vouchers with those of their sexual partners, fraudulently give their own voucher or the partner’s voucher to someone else, or simply choose to go to a different clinic for follow-up. Claims are quarantined or rejected if any of these potential errors are detected, but not until the claim has been processed by the TMA, and the fraudulent patient has already been treated. Since the original voucher is attached to the submitted claim, the clinics do not always have a way of verifying these external aspects of voucher validity. Although their direct involvement in the claims process is minimal, it is their identity that is often contested in the vetting process. G. Mobile Platform Developer (researchers) As ICTD researchers, we are of course also stakeholders the claims management process – initially as outside observers, later as designers interested in using technology to measurably improve the process, and finally as researchers interested in watching the mechanisms by which the process changes over the course of the project. From an outside perspective, our role is most allied with TMA, the technical partner in the project; however, since the aid agencies and FMA are interested in replicating the mobile device system in other OBA projects, there is a vested interest in the new technology from other stakeholders as well. V. D ESIGN Claim Mobile is a two-part system, including a web-based PHP/MySQL application and a Java-based mobile application running on Palm (GarnetOS) phones. For the pilot program, the web-based application has a single level for all users, but the final implementation will be tiered, having appropriate access levels for service providers, management agency users, medical advisors, etc. Both the web and the phone applications require user login to protect patient data. The two applications are paired, designed such that the phone-based application uploads claims to the web site, and downloads configuration information (drug lists, status feedback, claim form backups) from the web site. Eventually, the web application will also connect to the TMA database, sharing the cross-checked and validated claims form data directly so the TMA’s staff do not have to do redundant data entry. To facilitate end-user training, both of the applications are based on the original claim form and largely retain the same structure, titles, and information. Figure 5 illustrates some of the mappings between the phone-based user interface

and a revised version of the claim form. In addition, all of the codes and tables in the web application database also include mappings to their equivalents in the TMA database, so the information can easily be transferred between the two databases. A. Claim Mobile The web application is designed primarily with three functions in mind: claims submission, feedback/communications process, and in-clinic claims management. Claims Submission: This is the bulk of where the service providers will spend their time. In this case, we adopt common design strategies such as (1) using pre-filled checkboxes to reduce the amount of required text entry, (2) limiting answers to valid options to reduce coding errors (see QC01,02, 04-09, 27 in Table II), (3) downloading logistical data such as drug prices into the application to eliminate pricing errors, and (4) calculating dependent values such as expected claim amount to eliminate arithmetic errors and save time. However, we must counter-balance potential fraud by also introducing crosschecks that are not clarified explicitly. That is, to encourage accurate clinical reporting (as opposed to clinical reporting that has been “fiddled” to make electronic claim submission more convenient or favorable), providers are allowed to submit inconsistent claims but are warned that they should clarify any discrepancies from normal OBA treatment protocol. Closing the Feedback Loop: Based on our primary finding from the clinic surveys and follow-up interviews, we have also included the ability for the clinics to send queries to the management agencies about particular claims and to receive live updates on the a claim’s status (e.g., whether it has been approved, the amount for which a claim has been approved, and explanations why the full amount may not have been approved – see Figure 6). Any changes to a claim’s status are included in this annotation audit trail, and anyone with access to the claim can respond to and receive queries.

Fig. 6. On this screen the service provider can view the current status of their claim, as well as any annotations or feedback from the management agency made in response to their queries.

In-Clinic Claims Management: In the phone application, the service providers can also link between consultations and follow-up visits, as well as between client and partner visits, so they can easily check to see if the valid voucher is being


Fig. 5.

From paper form to mobile phone: a mapping of the mobile phone interface equivalents for each section of the claim form.

Fig. 7. This is the list of claims currently entered or downloaded to the mobile phone. The first number is the claim form number, followed by the patient name, and then a number indicating the current status of the claim. (1-unsubmitted, 2/3-under review, 4-preliminary approval, 5-quarantined, 6approved, 7-rejected)

used by a returning patient, and that treatment of a partner or during a follow-up matches the medical history. For new phone installations, or if the claims data is lost, the mobile application will automatically download all prior claims data from the web application. Future versions of the application will also include financial summaries, outpatient statistics, and other reports that may be useful to the clinics. B. Claim Mobile Web The web application, having been commissioned in the middle of the fieldwork in response to program office findings (see Section IV.C) has two main functions: receiving claims and displaying them for review. Much of the claims receipt is invisible to the web application user, and written as a backend for the mobile phone application. The login user for the mobile application and the web application is the same – and the login information

Fig. 8. Claim Mobile Web: the web-based view of the submitted claim forms is also based largely on the original claim form, split into several sections, and ending with annotations for the claim, and a form for adding additional annotations.

given to the mobile application is used to authenticate with the application when submitting claims and annotation data. There are three primary views in the web application. The claims list can be filtered by service provider and is modeled on the claim summary report (Figure 3). From each claim in the claim list, the user can either (1) click on the claim number to access the individual claim (and annotation/status update functions, see Figure 8) or (2) click on a patient name to view all consultations and follow-ups for the both the client and


partner associated with that particular voucher. This allows the viewer to correlate treatments, lab tests, and diagnoses across visits. C. The Modified Claims Process In the modified claims process, the service providers continue to complete and submit the claim forms. However, in order to receive faster payment, as well as the immediate crosschecked feedback from the phones on claims completion, they also enter the data on the mobile phones, submitting each claim to the web application as it is completed. Prior to claims submission the status of the claim is “Unsubmitted (1),” after which it can progress through a number of stages. The service provider can verify that a claim has been successfully received if the claim status has been updated to “Under Medical Review (2)” or “Under Administrative Review (3)” for medical advisor review or database cross-checking (validation of voucher) respectively. If a digital claim has been verified, the TMA will set its status to “Preliminary Approval (4)” until the paper form with the voucher has been received. Once preliminary approval has been received, the clinic can be paid. If no voucher is received, or the wrong voucher is attached to the form, then the preliminary payment is subtracted from the next month’s payment until the error is resolved. In the meantime, the service provider can view status updates as they are made to the web application and sent to the mobile phones, and can send annotations on each claim, which then appear in the web application and in the status update window (Figure 6) when they are received. VI. P RE -P ILOT R ESULTS Having detailed the claims submission process, our findings with respect to the various stakeholders in the OBA project, and the design of the Claim Mobile system, we now discuss some specific results from our user studies. The pre-pilot demonstration was designed to last one claims cycle, following one week’s worth of claims (submitted in parallel through Claim Mobile and on paper) for two clinics through the claims submission process. A total of 35 claims were submitted to the web application, including the full complement of 18 claims from the urban clinic, 12 out of the 86 paper claims from the rural clinic, and 5 additional claims from the urban clinic following the pre-pilot study. The claims from the rural clinic spanned August 9, 2008 to August 27, 2008. We observed three patient consultations during our two visits to this clinic, as well as the preparation of the summary sheets for the 86 claims, taking careful note of what the service provider verified on each form. Notably, although “syndrome” is a required field (see QC05 in Table II), it was left blank in almost all of the claim forms. In one case, the drug was entered correctly, but with the wrong reimbursement value, and in another case, a drug was entered, but no reimbursement was claimed either in the subtotal or the total. At the time of the claim approval process, they were not reimbursed for the drug, because it had not been claimed in the amount, although it had been listed. Another

inconsistency is in lab reporting – some lab tests require a value to indicate the result, and where not included, the data entry clerk just changes the lab test in the database to one which does not require a result. This is an error, which never gets communicated back to the service providers because only errors which accompany a payment change are reported in the claim summary sheet. The 18 claims from the urban clinic spanned dates from February 16, 2008 through August 25, 2008. During the process of simultaneously entering some of the claims into Claim Mobile with the service provider, we were able to identify some problems: missing personal information, missing next visit date, and wrong consultation fees. However, not all claims were entered into and reviewed via Claim Mobile, and, as can be seen from Figure 3, three paper claims were submitted with wrong consultation fees, an error that would not have occurred with an electronic submission. In addition, a fourth claim was submitted with the wrong voucher. We were able to catch this while entering the claim into the mobile phone, noting that the voucher number did not match the consultation type, but too late to change the submission and retrieve the correct voucher. As a result, the claim has been quarantined until the correct voucher is given to the program office in Mbarara.

Fig. 9. Rural clinic staff entering data from claim forms into two of the phones.

With regard to the digitally-submitted claims, we spent about a day training the staff in the rural clinic on how to submit the claims, and returned later to spend another day in training. They were very enthusiastic, and although only one person was actually responsible for submitting claims, they were all training each other (Figure 9). However after the the researchers’ departure they have still not submitted any claims. It is unclear whether this is from technical difficulties or lack of time to enter the claims into the phone. The urban clinic has continued to submit claims, with five new claims arriving in the two weeks since the pre-pilot study. An interesting outcome from our observation of the claims review process is that there are many errors that are made that affect the quality of the data, but are never communicated to the service provider, in part because they have no attached financial consequences. The annotation feature (see Section V.A) enables attachment of quarantine codes to any claims that were in error without affecting the payment of the claim, providing feedback to the service providers on how to better complete the claims in the future.


One concern that emerged from this proof-of-concept study, however, was with the reliability and the speed of Internet access in the TMA office where the claims processing was occurring. Accessing individual claims took a long time, and the online database was completely inaccessible when the Internet connection was down, which occurred infrequently but noticeably often. It may not be desirable for the claims submission process to introduce a dependency on Internet connectivity where it is unreliable. Unfortunately, the financial and claim review aspect of the pre-pilot was halted early due to administrative and political reasons, the result of which is that payments in the OBA program as a whole have been halted; so, while the technical feasibility of the system has been demonstrated, the logistical details are still in process. We found that while the TMA’s database entry staff were enthusiastic at the prospect of spending more time reviewing claims and less time doing just data entry, their participation in the pre-pilot was limited by two factors: the press of other claims that still needed to be processed, and pressure from the TMA to be secretive about the data being processed until certain political issues had been sorted out. VII. D ISCUSSION A. Understanding Delays in the System Delays can occur in a number of places in the claims process, not all of which can be accounted for by the introduction of mobile phones. However, there are three key bottlenecks: 1) the delay in the health clinic between when the health clinic sees the patient and when the claim is submitted, 2) the time it takes to process the claims, entering each on into the database, and 3) administration of feedback to the health clinics, especially in case of errors. Claim Mobile is able to address all three of these cases by 1) encouraging providers to submit claims as they see patients, 2) reducing the data entry burden through the use of digital claims, and 3) eliminating the possibility of a number of errors, and providing a digital feedback mechanism to supplement the infrequent in-person feedback. However, another source of delay is the administrative and political dynamics by which program administration halts, although health distributors continue to sell vouchers, and health clinics continue to see patients. During these times payments are delayed unexpectedly for undetermined lengths of time, as can be observed from the early termination of our pre-pilot study. Understanding this particular delay is key: the TMA halts the program by withholding information, specifically claim reports, from other stakeholders in the system. This is possible because the database is owned and controlled entirely by the TMA. What happens when another outside database is introduced, with independent control? In this case, the data was not so much the key as the data entry staff that were responsible for approving the claims and validating the vouchers. At the same time, it is unclear where Claim Mobile Web fits in with the political strategy of the TMA.

B. Information Poverty In addition to trying to address delays, we have also tried to address information asymmetry and information poverty within the system, identifying where stakeholders are disadvantaged by lack of information, or lack of tools with which to use the information. This past year’s move of the data processing from Mbarara to Kampala especially has further exacerbated the gap between the people that have the information and the people that can make use of it. While limited remedies have been made to rectify the situation, with a paper-based process, these remedies have been ineffectual, leaving the PMO without access to necessary claims data, including voucher numbers. Through extensive stakeholder interviews, in particular with the program office in Mbarara, we have identified the need for a transparently accessible database, with the ability to generate reports based on the submitted claims data. While control over access to the database is a key means by which the TMA asserts itself in the OBA project, this practice is highly disruptive to the OBA program, causing deep difficulties for the health clinics and the program office, rather than affecting the financiers or the FMA. The initial design of Claim Mobile, reflected the paperbased process, and directly submitted claims from the mobile phones in the health clinics to the TMA, bypassing the PMO entirely. In response to our findings, we developed Claim Mobile Web as a means of re-engaging the staff of the PMO in the mobile claims process. The integration of the new web application database is specifically meant facilitate resolution of information gaps, not only sharing the information with the people that can make use of it, but also giving them the tools they need to make sense of the information. Likewise, for the mobile-phone application, we also specifically do not design one-way system in which claims data is going out and only money returns. Instead, the claims data created in the clinic is also used within the clinic to help them improve patient care, as well as the accuracy of future claims. C. Related Work There have been a number of recent technical projects on the use of ICTs for healthcare in Africa [8], [9], [10], [5], and specifically on mobile devices for healthcare in Africa [3], [11], [12]. However, many of these projects are design-focused and technology-driven, reflecting on designing a working technological solution to complete a particular task, rather than reflecting on the role of the technology in the system and how various solutions or approaches might affect social processes within the system. Braa describes two action research projects to deploy the Health Information System Program (HISP) in Cuba [13] and in South Africa [8], using Actor Network Theory (ANT) to think about how human and non-human (e.g. documents, events, software, standards) interact. He specifically addresses the challenges of designing for the multiple levels of entities involved in district health information systems, able to compare deployments across Mozambique, India, South Africa,


and Cuba. However, these entirely government-based contexts are much more hierarchical than the highly disparate multiorganizational context described here. VIII. C ONCLUSIONS

the fieldwork and setting up this pilot, as well as their honest feedback on this paper. R EFERENCES
[1] Karen G. Cheng, Francisco Ernesto, and Khai N. Truong. Participant and interviewer attitudes toward handheld computers in the context of hiv/aids programs in sub-saharan africa. In CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 763–766, New York, NY, USA, 2008. ACM. [2] Cynthia Casas and William LaJoie. Voxiva: Case Study. December 2003. [3] Brian DeRenzi, Neal Lesh, Tapan Parikh, Clayton Sims, Werner Maokla, Mwajuma Chemba, Yuna Hamisi, David S Hellenberg, Marc Mitchell, and Gaetano Borriello. E-imci: Improving pediatric health care in lowincome countries. In CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 753– 762, New York, NY, USA, 2008. ACM. [4] E.S. Berner and M.J. Ball, Editors. Clinical Decision Support Systems: Theory and Practice. Springer-Verlag, 1998. [5] Henry Lucas. Information and communications technology for future health systems in developing countries. Social Science & Medicine, 66:2122–2132, May 2008. [6] ITC - e-Choupal. http://www.itcportal.com/sets/echoupal frameset.htm. [7] 2006 Venture Strategies and Mbarara University population survey. http: //www.oba-uganda.net. [8] Jørn Braa and Calle Hedberg. The Struggle for District-Based Health Information Systems in South Africa. The Information Society, pages 113 — 127, 2002. [9] Tessa Tan-Torres Edejer. Disseminating health information in developing countries: the role of the internet. BMJ (British Medical Journal), pages 797—800, 2000. [10] Rowena Luk, Melissa Ho, and Paul M. Aoki. Asynchronous remote medical consultation for Ghana. In CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 743–752, New York, NY, USA, 2008. ACM. [11] Tapan Parikh. Position Paper: Mobile Phones may be the Right Devices for Supporting Developing World Accessibility, but is the WWW the Right Service Delivery Model? In W4A at WWW2006. ACM, 2006. [12] Tapan Parikh and Edward D Lazowska. Designing an architecture for delivering mobile information services to the rural developing world. In Proceedings of WWW2006. ACM, 2006. [13] Jørn Braa, Ola Hodne Titlestad, and Johan Sæbø. Participatory Health Information Systems Development in Cuba the Challenge of Addressing Multiple Levels in a Centralized Setting. In Proceedings of Participatory Design Conference 2004. ACM, 2004. [14] Michael Demmer, Bowei Du, and Eric Brewer. Tierstore: A distributed file-system for challenged networks. In Proceedings of File and Storage Technologies (FAST), 2008.


There is clearly much additional work to be done, in which the lessons learned from this pre-pilot study will be applied in the development of a new version of Claim Mobile for a full pilot in Spring 2009. The outcomes from this study were three-fold. Firstly, the choice of the mobile phone as a platform was affirmed by the health clinics, for reasons of battery life, design for readability, portability (susceptibility to theft), and ease of data entry. Where we were concerned about introducing “qwerty” keyboards to novice users, our fears were alleviated, and all of our users assured us that “we can learn,” which they did, quickly. Secondly, the mobile platform is not a sufficient solution for this program, and alone has the potential to exacerbate information asymmetries between the stakeholders. To address this issue, we complement the mobile phone platform with a web application. However, Internet accessibility issues may require further development to enable local hosting and synchronization of the web application [10], [14]. Our final point is related – we consider the plethora of stakeholders in this project, and note that as technology providers we are not coming in as naturally neutral players. Our projects are necessarily disruptive, and equally potentially disrupted by other dynamics within the program as a whole. As a result it is necessary for us as researchers to position ourselves and our designs carefully, making sure to take into account the needs of all of the stakeholders, and not just our primary users. IX. ACKNOWLEDGMENT We thank Ben Bellows for his contributions to this project, and feedback on the paper. In addition, we would like to thank all the staff at our partnering organizations and in the health clinics for their assistance and patience in conducting


Computer Games in the Developing World: The Value of Non-Instrumental Engagement with ICTs, or Taking Play Seriously
Beth E. Kolko, Cynthia Putnam
 Abstract—This paper argues that it is important to study noninstrumental uses of ICT, including computer games. Specifically, the article presents the results of qualitative and quantitative work spanning eight years of investigation in Central Asia focused on computer gaming in public Internet cafes as well as private spaces. The results presented demonstrate that people do indeed play games in resource constrained environments. The paper demonstrates that games constitute a significant portion of the ICT ecology in developing regions and provide a pathway to people’s “first touch” of a computer, that gamers have more frequent interaction with technology than basic Internet users, that games bring more diverse users to computers by providing a pathway to ICT use for people with lower levels of education, and that games can motivate innovation in the technology space. Additionally, our findings indicate that both genders engage in game playing. The article makes the case that games can be a source of informal learning about ICT, and as such, games and gaming culture in the developing world merit further study. Index Terms—international development, Central Asia, ICTs, digital games, gender, Internet cafes, technology use, entertainment

online traffic absorbed by gaming related activity. There is no ranking of popular games in each country, or listing of most popular gaming cafes. There‟s no comparison of how much it costs to play Counterstrike or how many youth under the age of 15 are estimated to have consoles in their homes. The HDI, in short, ignores games as an element of information and communication technology infrastructure within a country. Games, in fact, are invisible to the UN, as they are to just about every non-governmental organization and multi-lateral organization that has worked to create information and communication related projects in the past decade or more. The goal of this paper is not, actually, to argue that the HDI should include computer games as a measure of development. However, the purpose is to establish that games lead to learning computer skills and that, indeed, games are played by people throughout the world. Games are in fact a pivotal piece of a country‟s computerization, how its population gains information and communication technology (ICT) related skills, and how ICTs themselves begin to diffuse in developing world contexts. II. GAMES, LEARNING, AND CULTURE

I. INTRODUCTION The Human Development Report is published by the United Nations every year. It‟s an amalgam of facts and figures about infrastructure, health, agriculture, policy and other characteristics of most countries in the world that provide an overview of a country‟s stage of development. The Human Development Index (HDI) is extracted from the report, and includes, among other things, measures for landlines per capita, Internet nodes, and mobile phones. The HDI, however, has no column for gamers per capita. Nor is there an entry for percentage of GDP spent on playing games, or amount of
Manuscript received September 22, 2008. This work was supported in part by U.S. National Science Foundation grants #0326101 and #0219350, and by a 2000 Fulbright Scholar award. B.E. Kolko is an Associate Professor with the Department of Human Centered Design & Engineering at the University of Washington, University of Washington, Seattle, WA, 98195 USA. (e-mail: bkolko@u.washington.edu). C. Putnam is a PhD candidate with the Department of Human Centered Design & Engineering at the University of Washington, University of Washington, Seattle, WA, 98195 USA. (e-mail: cyputnam@u.washington.edu).

There is a robust academic community dedicated to exploring the connection between games and learning, drawing heavily from both education and psychology. Games are an increasingly central topic in educational research, with scholars researching games as part of informal learning [28], key skills like collaboration that people learn while playing multi-player games [29], psychological and reaction time skills gained from games [36], and the creation of actual educational games designed to teach complex skills [6,7], [9,10] [30]. As a companion piece to academic work, significant attention and dollars have been targeted at exploring the issue of games and informal learning in the US (e.g. the MacArthur Foundation‟s 5-year, $50 million project on Digital Media and Learning and the Robert Wood Johnson Foundation‟s $8.25 million program in Health Games Research). There are also organizations dedicated to NGO-like activity around games, including the Serious Games initiative, Games for Change, and Games for Health; these groups focus on developing games with explicitly pro-social goals in mind. Much of the research in the games and learning area has focused on the learning that occurs while people engage with


games. Developing world initiatives include work by groups such as the South Africa-based Mindset Network which has developed mobile phone-based games to teach math skills to girls (Mathstermind and Fashion Network), and literacy and numeracy games for disadvantaged youth developed by Pratham in India. These specific pro-social gaming projects similarly focus on in-game content and what people can learn as a result of playing. This paper, however, is more interested in how engagement with games serves as an ICT usage entry point. In other words, not what do people learn because of a specific game, but, rather, what do they learn because of the specific activity of gaming. To that end, this paper focuses attention on the commercial game space and how it diffuses throughout and affects nascent computer users in developing regions. This question of effect can be situated within a theoretical framework tied to cultural theory that investigates issues of identity and agency. Generally, cultural theory provides another lens through which we can view the importance of digital artifacts that transcends their literal or functional meaning [32]. Again, in the most broad terms, cultural theorists such as Bruno Latour [35] provide a framework against which we can examine technological artifacts not for what they are, but for what they enable. That is, games and non-instrumental uses of technology are important in the developing world not (only) because they teach people to collaborate or improve language skills, but, as this paper will demonstrate, because they provide an alternative mechanism by which many people experience their first “touch” of a computer. Additionally, we found that games allow users with less education and English language skills to interact with computers, and that they foster innovation and creative engagement with technology.

locations, and it was administered in Russian and other regional/local languages. In addition to the general sampling scheme, a three-stage process was used to select respondents that included Probability Proportional to Size sample of Primary Sampling Units (PSUs); consecutive random sampling of households in each PSU, and selection of a household respondent using a Kish Grid method. The survey instrument was designed by a team of researchers from the University of Washington, pilot tested in each country in conjunction with local researchers, and then revised based on analysis of pilot data. BRIF Research Group, based in Kazakhstan, translated the survey instrument from English to other languages. The University of Washington team back translated the completed Russian translation. Likert scale measures and other question formats were developed in response to initial open-ended interviews, ethnographic field notes, pilot tests and research on performing surveys in postSoviet contexts. Several steps were taken to guarantee high quality fieldwork including: (1) approximately 30% of interviews were checked through a back visit to the respondent‟s home; (2) interviewers were trained through workshops and practiced in a pre-testing phase; and (3) statistical analysis of logical inconsistencies were double checked with the original paper questionnaires and eliminated if necessary. Given the low rate of current Internet penetration in Central Asia, the survey also focuses on pre-existing patterns of information use, information seeking behavior, and levels of trust in various producers and sources of information. Since Internet usage rates in the region tend to be low, the survey targets the general population rather than Internet users; the findings include overall attitudes towards technology from a wider audience rather than simply usage patterns from a much smaller group. B. Ethnography Ethnographic results are from the CAICT project as well as earlier fieldwork conducted by the first author during a sixmonth residence in the region in 2000. Fieldwork data collection methods follow standard ethnographic format for participant-observation and include field notes and photography. In addition, interview studies have been conducted with various groups to further investigate issues relevant to survey findings. Qualitative data reported in this paper are drawn from multiple separate studies conducted in 2005 and 2006 including interviews with Internet users, interviews with mobile phone users, interviews with computer gamers, and a design ethnography. In addition, ethnographic observations are based on field notes collected during separate data collection trips in 2000, 2002, 2003, 2004, 2005, and 2006. IV. BEYOND STEREOTYPES OF GAMES AND GAMERS Before addressing the specific ways in which games provide an important pathway to ICT engagement, it is important to deconstruct some of the stereotypes of games and

III. METHODOLOGY Our research is the result of longitudinal qualitative and quantitative work done as a component of a larger project on the effect of information technology on society. The Central Asia Information and Communication Technology Project (CAICT) is a multi-year study of ICTs. The project goals include investigation of how ICTs diffuse within societies, how cultural issues affect technology usage, and how patterns of trust and confidence in media and institutions change over time as technology diffuses in diverse communities. A. Survey The project includes a yearly survey of four countries in Central Asia: Kazakhstan, Kyrgyzstan, Tajikistan, and Uzbekistan. The quantitative results presented here are based on the nationwide survey of 1000 respondents in each of these four countries age 15 and older, administered in 2006, 2007, and 2008, for a total of 1000 respondents per year per country, and a total of 12,000 respondents overall. Survey sample was based on census information for age, gender, ethnicity, and geographic location as released by each country‟s government. The survey includes multiple urban and rural sampling


gamers. Recent research has begun to establish that gamers in developed countries are not all male and not all teenagers [3334], that in fact the audience for games is diverse in terms of gender and age [1][22-24]. Studies of complex multi-player gaming environments, for example, demonstrate that although male players outnumber female players, it is those female players who actually spend more time online playing [25-26]. Games themselves come in many flavors, including the violent shooting games that gain so much press attention, but they also include puzzle games and simulations. Game systems such as the Nintendo Wii or games with alternative input modes like Rock Band or Dance Dance Revolution have similarly changed the cultural activity of games and broadened the audiences to which they appeal [27]. While not identically diverse, gamers internationally also belie some stereotypes. While gamers are still more likely to be male and under 30, in our research population, female respondents and those over 30 were as likely to play games as they were to use the Internet, See Figure 1. This finding indicates that depending on the population segment, gaming is either a more used pathway to ICT use than the Internet, or it is at least equivalent.

Figure 2: Photograph of Starcraft competition announcement outside Kazakh game club

30% 25%

15% 10%

8% 7% 6%
5% 0%
Female Male Kyrgyzstan 8% Kazakhstan 18% Tajikistan 5% Uzbekistan 8%

4% 3%

2% 1%







Under 30

Over 30

Figure 3: Male and Female Gamer Populations in Central Asia, 2008 Internet cafes vary in quality of equipment and furniture. Some businesses will have recent equipment, flat panel LCD screens, and glossy office furniture; others have older PCs, CRT monitors, and plastic chairs. Often the ones the feel more like a business center are located in the center of the city and cater to a mixed clientele. However, businesses located in neighborhoods similarly can serve a diverse clientele, although not necessarily at the same time. And games exist in a context of ICT diffusion overall, with Internet growth creeping slowly and mobile use skyrocketing, see Figure 4.
70% 60%

Internet users





Figure 1: Demographics of gamers vs. Internet users V. SETTING THE STAGE: GAMING CONTEXT IN CENTRAL ASIA Public gaming centers in Central Asia meet many of the preconceptions of computer game culture, but they are also nuanced environments. The crowds of young boys that are seemingly ubiquitous are reminders of gamer stereotypes, although our survey results indicate that women do indeed play games (see Figure 3), although women tend to play at home rather than in public cafes. Many of the popular game titles are familiar, but the style of play and the mechanics of getting the games to work in different infrastructure environments are unfamiliar. Counterstrike (CS) and Starcraft are literally everywhere, and kiosks on the streets sell countless numbers of CS mods, the cd covers in English and Russian. Starcraft competitions can be found in many of the capitol cities in the region, see Figure 6, and the World Cyber Games are a coveted destination (and Central Asian countries have placed quite high in the World Cyber Games in recent years).

40% 30% 20% 10%


Mobile Users

Computer Users


Internet users










Figure 4 Relative Rates of ICT Usage 2006-2008


A. A Tale of Two Tajik Internet Cafes Plazma is an Internet café located in the center of Dushanbe, the capitol of Tajikistan. Plazma is on Ryudaki Street, the central boulevard in the city. It is located in a twostory standalone building, with a very large sign on the side advertising a bar and café on the first floor, and Internet center, computers, fax, IP telephone, and other services on the second floor. Plazma is typical of centrally located Internet centers in the cities of the region in that it offers a wide range of services. However, it is also a particularly good example of a business that has specifically not bifurcated into two separate establishments in order to accommodate both gamers and a growing population of Internet users. Plazma has, instead, blended the two business models and offers a central room with ten computers with Internet access arranged on long tables but separated by small tabletop dividers to allow privacy, 3 IP telephone booths, a central desk where items can be photocopied, text typed, faxes sent, and – very welcome in the Tajik summer – air conditioning. There is also a sign posted in each mini workstation providing some guidance about usage that reads in both Russian and English: “It is categorically not allowed to visit sites with pornographic photos and video materials. In case of visiting above mentioned sites, administration has a right to fine you from 80 up to 300 somani,” the equivalent in 2006 of US$23-86. The sign serves not just as instruction not to visit porn sites, but also as a not-so-subtle reminder that one‟s online activity is being constantly surveilled. What should be obvious, of course, is that the specific sites or what constitutes pornography is left open to interpretation. That signage echoes signs in Uzbekistan cafes that warn about “inappropriate content” which can be either pornographic or political. On the same floor, but in a separate glass enclosed room is the gaming center with 28 game stations preloaded with games such as Grand Theft Auto, Cossacks, Need for Speed, Call of Duty, FIFA, and Half Life/CS and equipped with high quality headsets. On the wall in the main room where the business services are offered is a large embossed sign that runs the length of the room that has the site‟s web address as well as their tagline in Russian and English: All Our Life is a Game. The manager reports that Counterstrike and Starcraft are among the most popular games at the café. During the school year the café is crowded, but during summer vacation students visit less often – presumably because they spend less time in the center of the city. The customer base for the gaming stations ranges from schoolchildren to adults, and the adults tend to call ahead to see if a station is free and what games are in session before they come to the café. Using the services in Plazma costs two somani an hour to play games and four somani an hour to use the Internet. The ratio – games costing half what Internet access costs – is in line with general trends in the region. As argued above, when Internet access is metered and slow, the price differential between playing games and browsing makes it increasingly attractive to see games as the open-ended entertainment activity associated with computer technology rather than aimless browsing or opportunistic link following.

Gigant 2 is located on the edge of Dushanbe in a residential neighborhood. It is located in a one-story strip of local businesses, There is a bread store on one side and a small dry goods market on the other. On the sidewalk in front are several women selling seasonal fruit. Gigant 2 has streetside signage advertising Internet services, prepaid IP telephone cards, DVD/VCD and VHS rentals. The café/club rents movies, burns DVDs and CDs, and offers multimedia services. There are 24 stations in the main room of Gigant. There is also a back room that is attached to the main room and separated only spatially. There are six stations in the back which serves as a VIP room and includes two couches and a large floor fan. Gigant is nominally air conditioned, and heavy plastic sheets hang as a barrier at the front door to keep the cooler air inside, but in the height of summer it is very warm inside. There are two IP telephone booths in the café, and a site-wide LAN for watching movies. Dozens of games were preloaded onto the machines, but Lineage, CS, Need for Speed, and Cossacks were reported to be most popular. The business remains open all night, and from 9pm to 6am the only services offered are Internet access and Lineage. CS was not offered during the overnight because it was too difficult to find many other players online during that time frame. Gigant has a three tired pricing structure, similar but not identical to Plazma, see Figure 8. Playing games costs one somani per hour, being “online” also costs one somani per hour, and “Internet” costs 2.5 somani per hour. The distinction between online and Internet is something that appears with some regularity in developing regions, and it is a further motivation for individuals to develop usage patterns that downplay the importance of open-ended web browsing. In the case of Gigant, online activities include chatting, but Internet means web browsing since ISPs often charge by the kilobyte. Under those circumstances, then, browsing can be extremely costly and, perhaps even more of an inhibiting factor, not particularly transparent in terms of expense.

Figure 5: 2006 pricing structure at Gigant 2: Games 1 somani, Online 1 somani, Internet 2.5 somani Gigant is a café that melds general Internet services and gaming into one space. In the afternoon, it was packed exclusively with schoolage boys. The manager asserted that in


the evening, when adults returned home from work, the clientele turned over. Indeed, around 6 pm, some of the boys drifted out, but increasingly adults – including women, began coming into the business. These two profiles capture many of the business environments where people play games in developing world contexts. Their public nature, idiosyncratic policies, and tiered pricing structures are common elements and evidence of variability across venues.

somewhat counter to assumptions made about users in resource-constrained environments, but the so-called frivolous uses of technology are, indeed, often what brings people through the door [31]. The sheer amount of game activity and gaming culture as revealed by longitudinal ethnographic work in Central Asia establishes the importance of games as a part of ICT ecology in this resource-constrained region. And when schools are not wired and home access rates are low, game cafes are likely places for people‟s “first touch” with computers. A. Nurturing public Internet access sites in early years: the role of games In the past several years, a number of studies of Internet cafes have demonstrated the importance of such public access sites for users in emerging markets and other developing regions; many of these studies have focused on NGOsponsored telecenters rather than commercial sites [45][8][[11-13][18][21]. Generally speaking, though, in the global south, access to the Internet is more often in public, shared space than it is in private homes, or even the workplace or school. There are several characteristics of public access usage, however, that mean people‟s relationship to the Internet develops a particular shape. [17] Public access generally brings an awareness that one‟s usage is economically constrained, metered by time and often by kilobyte, inhibiting the link-following that characterizes much Internet usage in broadband contexts. Internet cafes, for the purposes of this discussion, are publicly accessible commercial or noncommercial sites where people gain access to a variety of ICTs, including Internet, IP telephone, photocopying, faxing, etc. Many commercial sites augment their income by providing some café-like amenities, but often the name “Internet café” is conceptual only. But Internet cafes are also often the first places that ICTs come to communities in developing world contexts. Whether established by an NGO or a local entrepreneur, Internet cafes advertise novel services to a community that is largely technologically illiterate, and they then face the challenge of convincing people they have a need for services they do not know how to use. Indeed, the capabilities of the Internet are themselves somewhat opaque to individuals who have not been exposed to a new media infused environment, and games are often a way to first draw people through the door. The research conducted in Central Asia from 2000-2007 pointed to an evolutionary pattern for commercial public access sites. To illustrate, take the example of Tashkent, the capitol of Uzbekistan. Tashkent is a city of approximately three million; it was the fourth largest city in the Soviet Union. In December of 2000, there were twelve operating Internet access points in Tashkent. This list included two sponsored by NGOs or large multi-lateral organizations, one funded by a local cultural center, and nine commercial cafes – most of which did not actually serve food or drink. That number has since grown to the hundreds, and the actual number is impossible to estimate given the fact that many operate without being fully licensed by the government, signage can




Throughout this project, we have maintained a focus on the study of diverse users‟ information landscapes -- their ICT ecology. In other words, this work takes particular interest in how specific pieces of ICT form unique mosaics in different usage contexts, giving individuals and communities access to variable modes of interaction, communication, and information-seeking. What has become clear over the years is that games and gaming provide an important – if often overlooked – piece of this puzzle [15]. Often dismissed as irrelevant to capacity-building projects, overlooked as a measure of a nation‟s ICT sophistication, and prey to countless stereotypes about users, games are a largely invisible component of the ICT landscape in developing regions. However, from the standpoint of users gaining facility with ICTs, and allowing ICT-related industries to gain an early foothold, games play an important role as they are often the first attractor – the first ICT with stickiness – for novice users. In other words, games are an alternative pathway to some users‟ first „touch‟ of a computer. On a functional level (as well as philosophical level), ICTs mediate distance and time [2][3][19]. In a country such as Kenya or Cambodia, nations with poor road infrastructure, the ability to transfer information quickly and over large distances has a transformative effect on the kind of information environment within which people live. On the other hand, there is also the conceptual mediation ICTs play as they bring fractured components of the globalization narrative to small communities that might be otherwise have very limited exposure to outside media influences. In Central Asia, ICTs are a linchpin of economic development as well as a symbol of modernization. Computers in schools projects bring technology literacy to youth, e-government initiatives slowly bring elements of government online and into some version of transparency, and within certain professional sectors computers streamline practice and make international collaborations more effective. However, when such top-down initiatives are bracketed, and instead focus is placed on bottom-up patterns of technology diffusion, ICT usage follows a different narrative. Adoption and usage patterns that emphasize ICTs for communication, gaming, and entertainment emerge. From movies to games to ringtones, ICTs become attractive to everyday people for non-work purposes. This pattern runs


be missing or misleading, and there is no reliable central directory of all functioning businesses in the city. However, it is safe to state that the Internet business in Tashkent as of 2007 was thriving. Home access is growing, numerous ISPs offer local dial-up service, others offer DSL, and schools and businesses are increasingly getting Internet access. In 2000 (the early years of Internet diffusion), it was not immediately clear to individuals why they should spend about US$1 per hour to use something that had linguistic and technological barriers, particularly in a country where the average monthly salary was US$20-30. Interviews with novice users over the course of five months in 2000 pointed to such confusion and illustrated the cultural gaps that acted as additional barriers to entry for new ICT users. There was an absence of external information resources in people‟s lives, so using analogies to describe the Internet as, for example, like yellow pages or encyclopedias was not particularly resonant. Keyboards were often in English yet people spoke Uzbek or Russian and needed the Cyrillic alphabet. Operating systems and software were often in Russian which was an additional hurdle to Uzbek speakers (Uzbek is a Turkic language, not Slavic). Printer drivers didn‟t have the Uzbek character set, there was very limited content online that was relevant to local inhabitants, and keyboarding or typing skills were not common. Literacy rates themselves were quite high, although the multi-lingual nature of life could make reading and writing a bit more complicated depending on primary and secondary language ability. [14][16] However, because copyright laws were loosely if at all enforced, there was no shortage of entertainment media available. The latest games could be bought for about US$1 in kiosks on the street, and VCD movies were about the same price. At the same time, games could be played either standalone on individual computers or over a café-based LAN. Alternatively, movies could be watched on the computers by an individual or a group. Cafés developed a two-tiered pricing structure where it cost roughly half as much per hour to play games. What emerged, then, was a pattern of usage in these Internet cafes where the majority of customers came in to either use computers to type documents (also much cheaper than Internet access), or to play games. During the day, schoolchildren would come to play a variety of games, and often café owners would let the youngest children hang around and watch movies or play games if the café was otherwise empty. Playing games provided young people with their first experiences touching computers. Playing demystified the technology, allowed them to gain keyboarding skills, taught them how to navigate operating systems and GUI menus, and generally build comfort and facility with computers. In those early days, in 2000, it appeared that revenue from activities other than Internet usage allowed Internet cafes to remain open in their communities. Although it was impossible to get owners to disclose actual revenue figures, in conversation and observation over the course of six years of field visits, it became clear that for many businesses, the revenue generated by game players helped sustained these early adopter businesses. Consequently, when

a community member did need occasional access to the Internet, there was in fact a local site to visit. However, without games and other non-instrumental uses of the computers, the businesses would have had a much smaller regular customer base on which to draw. We would argue, then, based on our interviews and observations that games played a pivotal role in the ICT diffusion within that country. And once people started playing games, particularly LANbased games, they began chatting and utilizing other communication tools. Their usage of ICTs expanded beyond games, but it was the gaming activity that allowed them to enter the so-called information society. B. The appeal of games as a pathway to ICTs compared to other technologies While initially games might have been appealing because they were an inexpensive way for users to begin experimenting with computer technology, there was also a functionality argument in favor of them. In many developing world contexts, including Central Asia, Internet access tends to be fairly slow. Often an Internet café will have a dozen or more computers sharing one dial-up or DSL line. Such bandwidth limitations play out, for example, in a contemporary web page from a more developed country taking two or three or five minutes to load. In and of itself that may not seem a hurdle, but the time adds up. Using a basic free webmail application like Yahoo or Hotmail becomes an exercise in patience. To write an email in Yahoo mail, for example, takes about six page loads: one to load mail.yahoo.com; a second to enter a username and password and wait for login authentication, a third to click on the inbox and check mail; a fourth to click on compose to write a message; a fifth to click on send and wait for the message to go out; a sixth to click on check mail again and be returned to the inbox. Do the math, and suddenly it‟s 20-30 minutes to write one email – and that‟s without having to check someone‟s email address by loading the addressbook. Because these are public Internet sites, pop mail or SMTP is of limited utility, and so webmail quickly became the mail application of choice. However, people were forced to use webpages created in 2000 with access speeds more approximate to 1995. The Internet, then, was simply not perceived as terribly efficient; indeed, it was not at all efficient. Opportunistic web browsing was not a habit easily adopted when web pages took so long to load. Expensive and slow, the Internet was a window to frustration rather than a window to the world. Added to this limited utility, however, was a slightly more acute problem of surveillance and censorship.. Many Internet cafes had signs posted warning that accessing inappropriate content would result in fines or arrest. The managers or administrators would sit at a desk and monitor traffic; users were crowded elbow to elbow with very little privacy for the items on their screen. Inappropriate content was rarely if ever defined. Sometimes it was specified as political information, or inappropriate political information; other times pornography was banned. But what constituted inappropriate


political content could change; a website with acceptable regional news one day might be overly critical of the president the next and suddenly be on the banned list. The habit of censorship and surveillance for one‟s Internet usage provided another tamping mechanism for Internet use at the Internet cafes, and another motivation for those who wanted to learn about computers to gravitate toward games. Amidst all this gaming in the region – activity that was economically as well as socially appealing, gaming-specific centers gradually emerged. While game playing allowed Internet cafes to remain open in the early days of ICT diffusion, eventually the industry matured substantially enough that gaming centers split off from Internet cafes. There were enough Internet users producing Internet-only revenue that many businesses could focus on one or the other. Many crossover businesses remain, especially in countries with lower percentages of Internet users (Uzbekistan, Tajikistan), but one can see that as the sector matures and a business can take in enough revenue from Internet access to remain sustainable, that game cafes separate themselves from Internet centers. It is not a zero sum game, though, and as the numbers of Internet users grows, gaming remains a vibrant activity. Indeed, the presence of multiple gaming centers throughout the region – particularly in the capitol cities -- makes it clear that games are being played, both players and games are diverse, and gaming provides a social interaction platform.
70% 60% 50% 40% 30% 20% 10% 0%

we wanted to identify those users who overcame the barriers to entry for either activity. Our goal in this exploration was to investigate if the two groups differed, and if so, how they differed and what the differences would suggest about potential barriers. Two direct logistic regression analyses were performed using SPSS to predict gaming and Internet use outcome from eight predictors: (1) age; (2) gender; (3) years of schooling; (4) living in a rural or urban environment; (5) mobile phone use; (6) ability to speak and read English; (7) ability to speak and read Russian; and (8) socio-economic status reported on a scale of one to three. A. Comparing predictors of game playing to Internet use First, we analyzed game playing. A test of the full model with the set of predictors against the null model with no predictors was significant, 2(9, N=12000) = 2268.52, p<.001, Nagelkerke R2 = .317, indicating that the set of predictors reliably distinguishes between individuals who play games and those who do not. The approximate variance in predicting game playing accounted for by the set of predictors is 32%. According to the Wald criterion all eight variables reliably predicted computer gaming--listed here in order of influence: (1) age; (2) ability to speak and read Russian; (3) owning a mobile phone; (4) living in an urban environment, (5) gender; (6) ability to speak and read English; (7) years of education; and (8) SES. Next, we analyzed Internet use. A test of the full model with the set of predictors against the null model with no predictors was significant, 2(9, N=12000) = 2852.51, p<.001, Nagelkerke R2 = .447, indicating that the set of predictors reliably distinguishes between individuals who use the Internet and those who do not. Internet use accounted for by the set of predictors is a striking 45% According to the Wald criterion, all eight variables reliably predicted Internet use--listed here in order of influence: (1) ability to speak and read English; (2) age; (3) years of education; (4) owning a mobile phone; (5) living in an urban environment; (6) ability to speak and read Russian; (7) gender; and (8) SES.

Home ***
Kyrgyzstan N = 121 Kazakhstan N = 215

General InterFriend's Access net cafe house Point ** *** 6% 2% 10%

School ** 8% 10% 20%

Work 15% 12% 11%

49% 62% 27%

27% 5% 30%

15% 22% 17%

Uzbekistan N = 115 Tajikistan N = 122

Figure 6: Sites of Gaming in Central Asia, 2008
*** = p <.001; ** = p <. 05 Statistics based on Chi-square test


500.00 400.00 300.00 200.00 100.00 0.00 Age Gender Years of schooling 56.36 Living in an urban /rural

As discussed in the previous section, games can provide a mechanism by which youth can be drawn into ICT training centers, and games also provide a revenue stream for owners of public access ICT sites. Additionally, our survey results demonstrate that games offer tangible assistance in overcoming barriers to entry for novice users by allowing users with less education and English language skill to gain experience with ICTs. In order to explore potential barriers to entry for novice users we wanted to first identify what types of people play games compared to those that use the Internet, In other words,

Mobile phone use

English Russian




97.72 12.41

112.10 113.51



44.02 11.19

Internet Use 248.22

196.40 142.06 171.21 339.05 135.45

Figure 7: Predictor effects on the models according to the Wald Statistic


While the sets of predictors are the same, there are notable differences. First, the set of variables does a much better job at predicting Internet use than game players, indicating there is more homogeneity in the Internet using population; however, both models were very good at predicting the outcome. Second, the importance of individual variables is different. Whereas the ability to speak English is very important to predicting Internet use, it was less influential on game playing. Additionally, education level is very important to predicting Internet use, but not as much for computer gaming. Conversely, age is a stronger predictor of gaming. Together, these items suggest that entry into the game playing realm is more accessible to a different segment of the population. If the goal, then, is to broaden participation in ICTs, games provide a pathway to ICT usage for users who would not necessarily have the education or English language ability to become Internet users. See Figure 9. This finding is significant in that it clearly establishes that gaming and Internet use attract different segments of the population, and that an individual‟s likelihood to gravitate to one version of ICT over the other is not only due to availability or interest, but is also due to societal factors such as education. VIII. FINDINGS: GAMES AS A GATEWAY TO AND MOTIVATION FOR INNOVATION The strong attractor force of games can also be a motivator for innovation. Many games encourage users to become active participants in a digital environment, and the enthusiasm many players feel for games can motivate them to learn new technical skills in order to facilitate their playing. A. World of Warstan Blizzard‟s World of Warcraft (WoW) is a subscriptionbased game that requires Internet access. It is a massivelymultiplayer game that has thousands of players on each server. Much of the gameplay relies on collaboration, and players form guilds; trade happens in auction houses, and the game requires a certain critical mass of players to run effectively. The technology and banking infrastructure required by WoW would seem to relegate it to an impossible game in the context of Central Asia, a region with relatively slow Internet access and essentially no credit cards. However, in at least two of the capitol cities in the region, local ISPs have devised a way to offer WoW to their subscribers. Locally run servers allow DSL subscribers to play innetwork, so, for example, there is a Tashkent-wide version of World of Warcraft. The game is in Russian but with a server attached to the provider. Most of the people are from the capitol given the pattern of DSL diffusion. Players recruit friends and classmates, but because of the low in-game populations, the basic mechanics of the games are different from standard play in developed world contexts. For about UDS$12-13 per month, players get unlimited hours online with both WoW and ICQ. All other Internet activity is metered by kilobyte, and MSN and Yahoo Messenger are metered as well. As for WoW, there are only a few hundred users online in the world, which means there are

very few guilds, and the auction house is not a particularly effective mechanism for trading goods. While economically viable in this context, Azeroth, the world of WoW, is also a largely empty world for these players. However, clearly the online game has a draw, as do many other game genres. The local hacks that people use to be able to play games are a testament to the appeal that games of diverse genres have for users. B. Games as motivation to gain technical expertise Similar local adaptation can been seen among local gamers in Bishkek. A group of, neighbors, residents of the same large apartment building, discovered that they all liked to play the same games, but they preferred to play at home because they saw it as cheaper and they had “more freedom.” So they ran a LAN down the outside of their nine story Soviet style flat, connecting eight neighbors together so they could play together. Gamers display a fair amount of creativity in ensuring they are able to remain connected with global gaming culture. Their narratives of themselves as gamers, however, emphasize the public and social aspects of their gaming which is not necessarily consistent with gaming in the US, but does resonate with gaming culture in Korea and China. One of eight Kyrgyz gamers interviewed in 2006, Yuri echoed the pattern of games as a motivator for learning more about technology. He started playing games in the 4th grade, and was introduced to games by playing Flight Sim at a friend‟s house, long before computers had been introduced to his public school. After Flight Sim, he began going to computer clubs where he fell into the world of Counterstrike. Now 19, he plays games over a LAN only in the clubs because, as he says, Internet is too expensive. After playing for years, he gained a variety of computer-related skills and now works as an administrator at a game café. He calls his friends and arranges for them to come after hours and gather together to play. Although none of the other gamers interviewed in Kyrgyzstan in 2006 had direct stories of their gaming leading to jobs, they did all emphasize the collective, shared knowledge of their gaming circles, the enjoyment of gathering in public places together to compete, share, and improve skills. Their stories also stand in contrast to the usual ICTD narratives about how newcomers become first acquainted with ICTs and the associated information society. Often, development projects that incorporate technology emphasize instrumental uses of technology. Games, however, are part of an open-ended and alternative exploration of technology. IX. CONCLUSION Much of the research on game culture concerns itself primarily with gaming in extremely developed contexts – where resources are abundant, and where gamers‟ relationship with technology has evolved in the context of that abundance. Game culture is just as interesting, and just as vibrant, and just as reflective of complex social dynamics when it takes place in resource constrained environments and when the gamers who adopt these technologies have very different relationships


with the technology. Central Asia is one such resource constrained environment, and while the specifics of the region give it particular political and economic constraints, the stories that emerge of games‟ importance in allowing ICTs to take hold during early stages of technology diffusion, as well as the persistent public nature of gaming activity help to broaden the picture of what games can be and what purpose they can serve within an overall ICT ecology. It may also be that gaming culture is the non-instrumental use of ICT that can parallel some of the adoption stories for mobile phones. Games are fun, and there‟s nothing wrong with fun, even in a development context. If talking to friends and family is part of the leverage that gets people to adopt mobile phones, then playing with friends may be part of the leverage that gets young people to develop further skills with computers. As we have argued in this paper, gaming culture is vibrant and thriving, and it provides a potential “first touch” with ICTs , especially for people who may not have access to computers at home, work, or school. Additionally, our survey demonstrates that games provide different barriers to access than Internet usage, broadening the reach of ICTs. Finally, games and the enthusiasm that gamers have, can motivate innovation and technological skill acquisition. Ultimately, the goal of this paper has been to bring attention to games and some of their possibilities as an element of ICT ecology in resource constrained environments. Games have been almost completely ignored by the ICTD discourse, and as the community seeks to better understand how and why people adopt ICTs into their lives, it seems important to include entertainment-related uses of ICT as part of that adoption pathway if we hope to truly comprehend the cultural and economic changes associated with technology. ACKNOWLEDGMENTS Our thanks to Mark Chen, members of the CAICT research team, including Erica Johnson, Emma Rose, and Odina Salikhbaeva. Thanks also to our local researchers, research participants, and blind reviewers. REFERENCES
[1] [2] [3] [4] [5] Cassell J. and H. Jenkins (eds.), (1998). From Barbie to Mortal Kombat: Gender and Computer Games, Cambridge: The MIT Press. Castells, M (1996). The Rise of the Network Society, The Information Age: Economy, Society and Culture Vol. I. Blackwell: Cambridge, MA. Castells, Manuel (1997). The Power of Identity, The Information Age: Economy, Society and Culture Blackwell: Cambridge, MA Colle, R. (2005). Memo to Telecenter Planners. Electronic Journal of Information Systems in Developing Countries, 21(1), 1-13. Dagron, A. G. (2001). Prometheus Riding a Cadillac? Telecentres as the promised flame of knowledge. Journal of Development Communication: Special Issue on Telecentres 12(2). Dubbels, B. (2003) Video Games as Metaphor for Learning and Curriculum Design Egenfeldt-Nielsen, S. (2003). Review of the research on educational usage of games Fuchs, R. (1998). Little Engines that Did: Case Histories from the Global Telecentre Movement. Ottawa: IDRC. Garris , R. , & Ahlers, R. (2003). Games, motivation, and learning: A research and practice model

[6] [7] [8] [9]

[10] Gee, J. P. (2003) What video games have to teach us about learning and literacy. Palgrave/Macmillan: New York, NY [11] Hudson, H. E. (2006). From Rural Village to Global Village. Mahwah, New Jersey: Lawrence Erlbaum Associates, Publishers. [12] Jensen, M., & Esterhuysen, A. (2001). The Community Telecentre Cookbook For Africa - Recipes For Self-Sustainability. Paris: United Nations Educational Scientific and Cultural Organization. [13] Khelladi, Y. (2001). What Works:The Infocentros Telecenter Model. Washington DC: The World Resources Institute. [14] Kolko, B.E. (2002). “International IT Implementation Projects: Policy and Cultural Considerations.” Proceedings from the Annual IEEE IPCC Conference, Portland, OR, September 2002. 352-359. [15] Kolko. B.E., Thayer, A. (2003).“Games as Technological Entry Point: A Case Study of Uzbekistan.” Proceedings of the Digital Games Research Association. Utrecht University. 19 pages. [16] Kolko, B.E (2006) “Cultural Considerations in Internet Policy and Design: A Case Study from Central Asia.” Critical Cyberculture Studies: Current Terrains, Future Directions. Ed. David Silver and Adrienne Massanari. New York: NYU Press. 145-157. [17] Kolko, B.E., Rose, E.J., Johnson E. (2007). “Communication as Information-Seeking: The Case for Mobile Social Software for Developing Regions.” Proceedings of ACM World Wide Web Consortium Conference. 863-872. [18] Mercer, C. (2006). Telecentres and transformations: Modernizing Tanzania through the Internet. African Affairs, 105(419), 243-264. [19] Poster, M. (1990). The Mode of Information. University of Chicago Press. [20] Proenza, F. J. (2001). Telecenter Sustainability - Myths and Opportunities. Journal of Development Communications, 12. [21] Proenza, F. J., Bastidas-Buch, R., & Montero, G. (2001). Telecenters for Socioeconomic and Rural Development in Latin America and the Caribbean. Washington DC: Inter-American Development Bank. [22] Schott, Gareth R. and Horrell, Kirsty R. (2000). “Girl Gamers and Their Relationship with the Gaming Culture,” Convergence, v. 6, n.4, 36-53. [23] Taylor, T.L., Kolko, B.E. (2003). “Boundary Spaces: Majestic and the Uncertain Status of Knowledge, Community and Self in a Digital Age.” Information, Communication & Society. 6:4, 497-522. [24] Yates, Simeon J. and Littleton, Karen. (1999). "Understanding Computer Game Cultures: A Situated Approach," Information, Communication, & Society, 2:4. [25] Williams, D., M. Consalvo, S. Caplan & N. Yee. (2009, in press). Looking for gender (LFG): Gender roles and behaviors among online gamers. Journal of Communication. [26] Williams, D., N. Yee & S. Caplan (2008). Who Plays, How Much, and Why? A Behavioral Player Census of Virtual World. Journal of Computer Mediated Communication. [27] Nadia Bianchi-Berthouze, Whan Woong Kim, Patel Darshak, "Does body movement engage you more in digital game play? And Why?", Proceedings of the Int. Conf. of Affective Computing and Intelligent Interaction, LNCS 4738, 102-113, Lisboa (Portugal), September 2007 [28] Stevens, R., Satwicz, T., & McCarthy, L. (2007). In-Game, In-Room, InWorld: Reconnecting Video Game Play to the Rest of Kids' Lives. In Katie Salen (Ed.), The Ecology of Games: Connecting Youth, Games, and Learning (pp. 41 - 66). Cambridge: The MIT Press. [29] Nardi, B. and Harris, J. (2006). Strangers and friends: collaborative play in World of Warcraft. In CSCW '06: Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, pages 149-158, New York, NY, USA. ACM Press. [30] Holland, W., Jenkins, H. & Squire, K. Theory by Design (2003). In Perron, B., and Wolf, M. (Eds). Video game theory reader. (pp. 25-46). London: Routledge. [31] Sandvig, C. (2006). The Internet at Play: Child Users of Public Internet Connections. Journal of Computer-Mediated Communication, 11 (4). [32] Wise, John MacGregor. (1997). Exploring Technology and Social Space. Thousand Oaks, CA: Sage. [33] Lenhart, Amanda, Kahne; Middaugh, Ellen, Rankin Macgill, Alexandra; Evans, Chris; Vitak, Jessica. (2008). “Teens, Video Games and Civics: Teens' gaming experiences are diverse and include significant social interaction and civic engagement”. Pew Internet and American Life report. September 16, 2008. Available at http://www.pewInternet.org/pdfs/PIP_Teens_Games_and_Civics_Repor t_FINAL.pdf [34] Lenhart, Amanda; Jones, Sydney; Rankin Macgill, Alexandra. (2008). “Adults and Video Games” Pew Internet and American Life Project

Data Memo. December 7, 2008. Available at http://www.pewInternet.org/pdfs/PIP_Adult_gaming_memo.pdf. [35] Bruno Latour. (2005). Reassembling the social: an introduction to Actor-network theory, Oxford, New York: Oxford UP. [36] C. Shawn Green andDaphne Bavelier. (2003). “Action video game modifies visual selective attention.” Nature. Vol 423. pp. 534-537.


Content Creation and Dissemination by-and-for Users in Rural Areas
Sheetal K Agarwal, Arun Kumar, Amit Anil Nanavati, Nitendra Rajput
IBM Research, India Research Lab 4, Block C, ISID Campus, Vasant Kunj, New Delhi, India. Email: sheetaga,kkarun,namit,rnitendra@in.ibm.com

Abstract—83% of the world population does not have access to Internet. Therefore there is a need for a simple and affordable interaction technology that can enable easy content creation and dissemination for this population. In this paper, we present the design, development and usage pattern of a VoiKiosk system that provides a voice-based kiosk solution for people in rural areas. This system is accessible by phone and thus meets the affordability and low literacy requirements. We present usability results gathered from usage by more than 900 villagers during four month of the on-field deployment of the system. The on-field experiments suggest the importance of locally created content in their own language for this population. The system provides interesting insights about the manner in which this community can create and manage information. Based on the use of the system in the four months, the VoiKiosk also suggests a mechanism to enable social networking for the rural population.

information that caters to a wider section of the population, thus leaving out the details for specific villages. One of the common methods of providing local information (such as a health-related warning) is done through announcements using loud speakers in vehicles. Figure 1 shows one such vehicle being used by a government healthcare official to announce an upcoming health camp in the village.

I. I NTRODUCTION The total rural population in 2005 is about 51% worldwide. If we look at less-developed nations, then this number is even larger, at 57%, which translates to more than 3 billion people. For India, the percentage of rural population is even higher, 71%. Based on the projections in [1], this number is not likely to change fast in the coming years (the percentage yearly change from rural to urban is projected to be around 1%). Even so, most of the HCI research has been focussed on the interaction with computers, to which this population does not have access. Therefore it is extremely interesting to look at sections of this large population base and identify the information and communication needs, the means to achieve these needs, and study the interaction modalities of this society. A majority of this 71% of rural population in India is illiterate or at least textually-illiterate [2]. Therefore though they can count currency and make phone calls using numbers, the ability to read or write is very limited. The means of interaction of this society is either completely voice-based or paper-based. Some people (esp. the money lenders, shopkeepers) make paper notes of their business. Most people (such as farmers) interact using voice, and get information through radio, television and other public announcement methods in villages. News papers continue to be another source of accessing information. However the common problem across these information sources for these villagers is the lack of locally relevant content. Most news papers, radio and television programs provide



IBM India Research Lab

June 24, 2008

Fig. 1.

Announcement of a government organised health camp.

There is a lack of enough information sources that can create sufficient locally relevant content such as doctor visit organised by the government, change in school timings, delay of the train service, list of movies in the village community, etc. If there is a system that can enable this community to create their own content, the information and communication ecosystem can be significantly improved. Since not much is studied about the rural population, to build an information system that can be used by rural population to create and access content, it is extremely important to follow the following design steps: 1) Perform a needs-study to identify the information needs of this population. It is important to understand what does a community consider as information. To enable easy access of information, a meaningful categorisation is also needed. 2) Follow a participatory design process to develop a lowcost, low-literacy solution. Since a majority of this rural population in India lives below USD 2 per day [3], it


is important that the solution is affordable. Moreover, the low literacy considerations also need to be taken in developing the end-user interface of this system. Participatory design process is important to study a population that has not been exposed to interaction modalities beyond paper, radio and TV. 3) Observe usage pattern of the solution for the rural population. Since not much research has been performed in enabling local content creation, the usage pattern of this population will be interesting to observe and can provide important insights. In this paper, we follow these three principals to build VoiKiosk – a system for increasing reachability of information kiosks in developing regions. A VoiKiosk is a voice based service available on the Telecom network [4]. As opposed to PCs, the phone penetration in rural India has been significant and continues to grow. Also based on field studies and literature reviews semi-literate and illiterate people are more comfortable with speech-based interfaces to access information services [5]. Leveraging the increased mobile penetration and comfort with speech based interfaces, we present this alternate model to create and host voice-based kiosks (VoiKiosks). We identify the information needs of villagers (Section III) in south of India by visiting several villages and talking to villagers and officials of an NGO that operates in these villages for more than seven years. Based on the needs, we implement the VoiKiosk system (Section IV) and involve a limited number of users and the NGO (Section V) in the participatory design phase. We deploy the system live in one village and gather important usage statistics by four months of live field deployment of the system (Section VI). We conclude the paper by discussing important insights gained during the entire process of needs-gathering, participatory design, prototype testing and usage analysis. II. R ELATED W ORK Several studies have suggested that rural communities have very different information and communication needs and patterns [6] [7]. Moreover, it has also been studied that technological innovation opportunities in rural areas have not been studied in great depth [8]. In [8], the authors present field studies suggesting that use of a local stakeholder contributes significantly to a better design of the technologies for rural communities. While there are significant design challenges for the rural community, at the same time, the potential of technology to improve their lifestyle is huge [9]. This forms the motivation for the work presented in this paper. A. Information Access and Dissemination In the current world of globalization, access to information is key to the development of a society [10]. The increasing amount of information on the WWW [11] is a reflection of this fact. Not surprisingly, therefore, there have been several efforts to provide this information to people in rural areas. There are solutions in the network space to provide low-cost Internet access access for rural areas [12], [13]. Significant work has

also been done on improving the usability for mobile Internet interaction. In [14], the authors highlight the various usability issues by studying the different critical comments by mobile Internet users. Despite all these solutions, the usefulness of Internet information for people in developing countries is very limited due to the fact that the content that is relevant for a villager is not available on the Internet [15]. This paper positions itself in the space of creation and dissemination of locally relevant content and is supportive of the statement: access to relevant information is key to the development of a society. B. Alternate Models for Local Information Systems Community Radio systems have been studied as a alternative solution to provide information in rural areas. In [16], the authors present a study that evaluates the effectiveness of 10 community radio stations in different rural communities in South Africa and report the popularity of programs. A sustainability study of the community radio service is presented in [17] for Nepalese radio stations. Extensions to community radio service to involve audience participation have been presented in [18]. Among other things, the success of community radio lies in the availability of relevant content for the rural communities. The CAM system [19] provides an interface to capture local data using the mobile phone camera. It has been applied for microfinance data capture [20] and for procurement and quality control in a rural cooperative [21]. Using camera phone to capture paper content is another approach to create local content that can be used in several application domains. We derive the importance of relevant content presented in the cited work and use these concepts in this paper. C. Speech and Mobile based Systems for Low-literacy Users Since textual literacy is a major issue in developing countries, especially in rural areas, several interesting information systems for this community are speech based. The community radio systems mentioned above are completely based on audio information. Studies from a spoken dialog system for rural India [22] show that users are able to navigate the dialog system irrespective of their literacy levels. A speech-based system for providing health information to low-literate users has been presented in [23]. This paper reports that there is a need to educate the users by creating a video to explain and demonstrate the system for a new user. Information systems based on low-end mobile devices provide an affordable alternative to PC-based systems. A system for providing information to farmers using mobiles has been presented in [24]. The authors in [7] conclude that exploring mobile social software holds significant potential as an information tool in rural areas. The MobilED [25] and the AudioWiki [26] are recent solutions that address this space by providing information to this population through mobile devices. The work presented in this section leads us to believe that creation and dissemination of locally relevant content through


speech interface using mobile devices can be used as an effective system to provide information needs of the rural population in developing countries. System level technology development work in this domain has been presented in [27] and [28] where the authors present technologies for content creation and linking. A solution based on these technologies [29] proposes to organize the unorganized urban poor businesses. In this paper, we extend the concepts of the World Wide Telecom Web [30] and apply them for providing information solutions to the rural population in developing countries. III. N EEDS S URVEY In order to understand the information needs of a community, it is important to know the community for a significantly long time. Selective interviews are often restrictive and may not provide deep insights that can be gained with time. The understanding of the working and living methods can either be learned through an extensive ethnography study or by learning from experience of organizations who have done such study and have a presence in the community [8]. We relied on the latter approach. This was one of the reasons for us choose villages in the Andhra Pradesh state in southern part of India. A. The NGO Operations in the Field Byrraju Foundation [31] is an NGO that has been operating in 198 villages in 6 districts of Andhra Pradesh. The NGO has performed detailed studies to identify the needs of the villagers from a much broader perspective. Their operations cover basic amenities such as sanitation, drinking-water and healthcare at one end to advanced services such as distanceeducation, remote-healthcare and rural BPOs. Following are the main activities of the NGO in these villages:

The NGO has a Village Coordination Officer (VCO) for every village. A VCO is usually a woman who has had 12 years of formal education. The VCO can only understand local language, Telugu. For every 3-5 villages, the NGO has positioned a Nodal Coordination Officer (NCO) to oversee activities in these villages. The NCO can understand English and is a degree holder in any subject. The NCO can operate the computer. Every village has a Gram Vikas Samitee (GVS) that is a committee of 18 members from the village who have volunteered to work with the NGO on the 18 different modules. B. Needs Gathering on the Field We visited four villages accompanied with the NGO field staff to understand the information needs of this community. The NGO staff provided the development history and details of the community for these villages. These four villages were identified based on the varying demographics to get a broader perspective. The first village (Vandram) has agriculture as the main source of livelihood and paddy is the main crop. Juvvala Palam is one of the larger villages and is geographically located as the central hub for nearly 4 lakh population in nearby villages. This village has a large population base that deals with renting of transport vehicles and so the associated business of mechanics, drivers is also very common. The third village (Cherukumilli) is primarily aquaculture based and has a number of fish ponds where aqua farmers cultivate fishes. Ibhimavaram is a very wealthy village where farmers have large lands and they usually employ labourers to cultivate their farms. Except for Cherukumilli, all the other three villages had a computer center (called as Ashwini Center). The chart in Table I summarises the important inputs gathered in the field. Each meeting in a village was with the GVS committee head, the NCO, the VCO and some members of the GVS committee of that village, all in one room. Most villagers share a phone within their family. Thus the cell-phone penetration rates are more than 50% considering the number of families that have a cell-phone. Ibhimavaram has 100% cell-phone penetration. Privacy of contacts or calls is not a concern in this open community. This culture enables easy sharing of cell-phones between family and friends. Almost all villagers mentioned that the mode of interaction with the cell-phone device is voice. People do not use text messaging, even though this is relatively cheaper. All cell-phones have Roman letter inscriptions and there is no local language support either on the keypad or on screen. All cell-phone users are number-literate and can recognize English language digits. However very few can understand the roman script for characters. The cell-phone signal is sufficiently strong in all remote areas. Although there are lot of electricity blackouts, but these intervals are short in duration, thus charging of cell-phones is not an issue. There are two major service providers (Airtel, BSNL), however there are about 5 different service providers available. Most officials and village committee members mentioned that people will be interested in knowing several schedule

Education: The NGO adopt one primary and one secondary government school in every village. They provide training to teachers, conducts distance education courses, and conduct health check-ups for students. Healthcare: There is a health center for every village where a doctor visits for 2 hours daily. The Village Coordination Officer (explained later) acts as an assistant to the doctor. There is an ambulance service for all the villages. Sanitation: The NGO has facilitated garbage collection system for all villages. The garbage collector generates and sells manure from the garbage to make the system self-sustainable. Drinking water plants have been set in some villages, they provide door-delivery of water at 10 paise per liter (USD x per gallon). Ashwini Centers: 20 villages have computer centers. These are used to provide distance education to students at school times. Later they are used to provide painting and embroidery classes to women. Agriculture/Aquaculture Advice: Experts in distant locations provide personal advice to agri/aqua farmers based on the crop/fish conditions.










Main Operators Airtel/ BSNL/ Idea/ Vodafone Airtel/ Tata/ BSNL/ Vodafone Airtel/ Idea


Will pay?

Information Required


Juvvala Palam




Depends on value of information. Voice is better

Not initially.

Program schedule of Ashwini, Information categorised on Modules, Pesticides, Medicines, Aqua information, sports updates, railway enquiry. as Community news, agriculture info, NO to train, no to Electricity, doctor visit, Teleconferencing information announcement, panchayat meetings, ankur channel program schedule. Hub village for 4 lakh population in nearby villages. information about village, Panchayat meetings, GVS meetings

Yes for pilot

same above

Yes for Pilot









Idea/ Airtel


After realising the value, they may. Not initially. Yes

Tractor renting, electricians, etc. panchayat meeting, national fish rates, V-agri, V-agri, v-agri

GVS very interested for pilot.

related information such as distance education class schedule, village committee meting schedule and minutes. Other information needs were based on agriculture information, community news and crop rates. For the Juvvala Palam village, people were interested in knowing about electricians, mechanics and different rental options for farming equipments. Except for Ibhimavaram, other villagers mentioned that the villagers may not pay for the service initially, but once they can see value from the information, they have the capacity to pay for the information. Based on these meetings, we identified the following categories for which the information is required by villagers:

utes, sports achievements, etc. It was clear from the on-field interactions that a phonebased information portal that enables interaction through voice will be ideal from the end-user perspective. Such a system will ensure that the end-users will not need to learn any new interaction modality nor a new device. Given the significant cell-phone penetration, a cell-phone based system can be used to fulfill the information needs of these villages. IV. P ROTOTYPE

• •

• • •

V-Agri: Provide agriculture and aquaculture advice to farmers on cell-phones. Job work: items and type required by merchants to that villagers can get business. Availability of transport. Health: Mostly static info such as doctor timings, date for the eye camp, student camp Aqua prices: Traders can upload prices of fish, prawns at which they will purchase. People: Static information about electricians, mechanics, carpenters of that area with phone numbers Entertainment: Name of movies in the nearby theater. Ashwini program schedule General info about the village: Population, availability of health center, schools, famous local food, etc. Community news: GVS meeting timings, meeting min-

Of the 10 different categories of information identified in the previous section, we built a prototype of the VoiKiosk system for four main categories. A VoiKiosk is a specific instance of a VoiceSite that can be used by villagers to create and access locally relevant content. A VoiceSite can be thought of as a parallel to a Website, but which can be accessed by dialing a phone number and information can be listened rather than being read or seen. Creation of a VoiceSite is made easy by the VoiGen system [27] through which anyone can call up the VoiGen system and interact with it through voice. This can enable any illiterate person to create her VoiceSite. Such a system enables easy local-content creation. All information in the VoiceSite is stored as audio messages that are recorded by making a phone call to the system. Since a VoiKiosk is a VoiceSite for the entire village, different type of users can update content in different sections. We explain the type of users and their interaction with the VoiKiosk in the next section. The kiosk operator (typically a NCO or a VCO) navigates through the VoiKiosk application


to configure it to offer various services for the village. Figure 4 shows a sample operational scenario for the VoiKiosk.
Health Message Health Message Health Message Health Message

Welcome Message

Create: Listen:

Operator All



Kiosk-operator uploading information on VoiKiosk


Ashwini Center Schedule

Agri/Aqua-culture Advice




Villagers calling the VoiKiosk for information

VoiGen allows creation and configuration of VoiKiosk

Create: Listen:

Operator All

Operator All

Expert Farmer

All All

Fig. 3.

Information on VoiKiosk: categories and access

Fig. 2.

VoiKiosk Usage Scenario

A VoiKiosk act as information and service portal for a village. It can be a central point of access for a community where information relevant to the community can be posted and accessed directly by the users themselves. This solution doesn’t rely on Internet connectivity which is most often not available in the rural areas and most importantly, it allows end-users to directly interact with the services, thus removing the dependence on the kiosk operator. Following are the four main categories of information available on the deployed VoiKiosk system for the village:

V-Agri: Farmers use this service to consult agriculture experts regarding their crop related problems. Currently a picture of the crop is taken and sent to an expert, who then sends a reply back to the farmer through the foundation. The turn-around time for this process is 24 hours. With VoiKiosk, the expert will be able to post his advice for the farmer on the VoiKiosk, reducing the turn-around time to 4 hours. VoiKiosk identifies farmers based on their caller IDs. Health Information: Information related to different health advisory and health camps is posted in the VoiKiosk. The schedule of the doctor visit to the health center is also posted on the VoiKiosk. The VCO can change the message if there is a change in the doctor visit timing. Ashwini Center Schedule: Information regarding new programs, schedule of daily classes or changes in the schedule are advertised by word of mouth, local newspapers or posters that are pasted at various spots in the village. Often people go to the the ashwini centers to get this information in person. The VoiKiosk has a Ashwini Center section where a kiosk operator can post the latest news related to the distance education program. Professional Services: In this section users will be able to record their personal advertisements. In the current practice, micro-businesses such as mechanics, drivers, daily wage skilled laborers use word of mouth advertising to reach out to clients. VoiKiosk provides an opportunity to increase their client base and increase business opportunities for them. Users call the VoiKiosk and record their advertisement which other villagers can browse by calling the VoiKiosk.

As shown in Figure 3, the information in these four categories can be modified by different kind of users. All information is accessible to everyone, except for the expert advice, which is specific to a particular farmer. We discuss about the different types of users in more detail in the next section. The call-flow for a specific interaction of the Kiosk operator with the VoiKiosk is shown in Figure 4. The Kiosk operator can choose to modify the welcome message for the VoiKiosk system, or modify information in any of the four different categories. Within a category, the operator is allowed to create a new information message, delete any existing message or rerecord an existing message.
Kiosk Operator

Record welcome message

Edit Edit to edit the welcome message Services to configure services

Select a service to add: Health, Agriculture, Weather, Transport Agriculture Record service description


Add to add information, Edit to edit the service description, Configure to configure the service, Back to go to service menu Configure Add an agri expert Browse registered experts Back to go to service menu



Fig. 4.

VoiKiosk Usage Scenario




Given the four categories of information, based on the creator and the consumer of this information, there are four types of users of the VoiKiosk system:


The Kiosk Operator has the permission to modify any content on the VoiKiosk. He can call the VoiKiosk from his cell-phone and the system identifies his caller-id and provides the right permissions. • Experts are allowed to post expert comments on agriculture and aquaculture for specific farmers. They dial the phone number of the farmer for whom they have to post a message. Experts can not post comments in any other category. • Farmers call in the system to listen to any message from experts. They can listen to information in other categories too. • Villagers can post an advertisements, or listen to any other general information in the VoiKiosk. A kiosk operator is usually identified for the VoiKiosk. The kiosk operator has permissions to add different categories, and also to add and delete information in any category. We have developed a kiosk template for a group of villages in Andhra Pradesh.

cognitive load on the users is less if they are told what all they can say to the system.

Fig. 5.

Initial prototype testing with the farmers.

A. Participatory Design Modifications We developed the prototype system in consultation with the NGO and then took this to the field to get feedback from the four different type of end-users. Since the Kiosk Operator has the maximum number of options available on the VoiKiosk, we started our testing with 2 Kiosk Operators. The initial design of the system was such that when the system expected a user input, there would be a beep sound at the end of the system utterance. However the beep sound was present only when the system used to record user voice and not at the time when the system would do a speech recognition on the voice. Since this fact was not known to the users, they did not know when to start speaking. Sometimes they would respond too early and sometimes they were too late. This resulted in increased frustration and so to make sure that they are speaking in the interval when the system accepts their input, they kept on repeating the same utterance, such as “exit, exit, exit”. We modified the VoiKiosk interface so that after every system input, there was a beep sound. The users could then figure out that they are always supposed to wait for a beep sound before they respond. This improved the system performance significantly. There was a system prompt that asked the following from the users: “Do you want information on Health or Agriculture or Ashwini Center Schedule or you want to about the people in this village?”. We expected that people will respond by saying either of Health, Agriculture, Ashwini Center or People. However the users used to answer Yes or No. So we had to change the prompt to “Please say Health to know about health related information, or say Agriculture to ...”. Although these descriptive prompts are too lengthy and not so natural, but this community preferred to listen to prompts that clearly tell the user what to say. This was a very good insight from these participatory design experiments. Users in village have more patience and prefer simple descriptive system prompts as compared to more natural sounding prompts. Perhaps the

From the experts perspective, they wanted that any advice they post to the farmers should also be archived in the NGO IT infrastructure. The NGO believes that this database of specific advices can act as a knowledge resource in the future. We created a FTP site so that all the content in the V-Agri section would be pushed to the NGO server at midnight. The duration of any message to be posted on the VoiKiosk was restricted to 10 seconds in the initial design of the system. The system automatically detects silence and identifies the end of the utterance. However if there is some noise in the background, then the system is not able to detect the silence and the message recording continues. This can result in user frustration, both during message recording as well at the time of listening. The other option was to ask the users to press a specific key to explicitly let the system know the end of utterance. But this would involves a key input, which we wanted to minimize, given the profile of target users. So we had fixed the duration to 10 seconds. However this duration was too less to record some of the messages from the experts. Since experts were more advanced users in terms of the device familiarity, they could use the keypad while talking over the phone. So we increased the duration to 30 secs for the Experts. They now have to press the # key to indicate the end of the utterance in case of background noise. The participatory design process improved the system significantly. With the first version of the system, the Kiosk operator used to take about 10 minutes to reach the Ashwini Center part of the VoiKiosk and then update a message. Now it takes him (and any novice user) less than 2 minutes to update a message. VI. U SABILITY


The VoiKiosk was tested with the villagers for a period of four months. We present the usage statistics of the VoiKiosk system by analysing the data through two approaches. The first approach presents the analysis by looking at the data as a


whole. This analysis is presented to demonstrate the increasing acceptability of the VoiKiosk system in the village. In the second approach, we present the results on the backdrop of the changes that were made in the system over time. This will provide more details about the effects of the usability of the system with improved interactions. In this section, we will present the two results in more detail. A. User Background and Village Demographics The pilot was deployed for villagers in the Juvvala Palam village in South India. This village has a population of about 4000 people that form about 850 families. About 70% of these families have a cell-phone. The male/female ratio is 100/94 in this village. The main occupation of this village is agriculture, especially paddy crop. Transportation is the main business for this village. People rent their trucks, tractors and smaller vehicles for a few days to people in nearby villages. These are used by farmers in their field and for a specific social function such as a wedding. The village a health center where the doctor visits for two hours every day. There is one bank in the village where about 10% of the families have a bank account. The average monthly income for a family is roughly about 2000 rupees (USD 50). All cellphone users are comfortable dialing numbers and talking with other people. However the use of text-message is very low. As a rough estimate, only about 10% of villagers use text-message for communication on cellphone. This is due to the fact that the local language (Telugu) is not supported on the device and since only about 20% of people know very limited English, text-messaging is almost non-existent. The village has a secondary school and a primary school. The NGO facilitates English and Math classes through a distance education program in these schools. The NGO also facilitates distance education of MS Office, spoken English, embroidery and painting classes for adults. Villagers from the Juvvala Palam village participated in the pilot. B. Overall System Usage The VoiKiosk system was live, 24 hours a day and all seven days of the week for four months. In these four months, the system received a total of 20499 calls from 976 villagers. We assume that a unique phone number would map to a specific villager because the trace to users is based on the caller identification in the VoiKiosk system. Table II shows the number of calls that went to the different services in the VoiKiosk. The advertisements section of the VoiKiosk was accessed maximum number of times. The third column in this paper shows the percent of the total number of calls that access this service. Since this number does not add to 100, not every call accesses a service. Users spent a total of 477 hours on the VoiKiosk system. The average call time was 83 seconds, with a maximum of 49 minutes and minimum of zero seconds. The number of calls have been increasing since the pilot was launched. The graph in Figure 6 hints at the increased acceptability of the VoiKiosk system over time. On an average,

TABLE II S UMMARY OF DIFFERENT SECTIONS VISITED BY USERS . Description Agriculture Health Ashwini Center Advertisements Number 1640 2383 1749 7492 Percentage 8% 11.6% 8.5% 36.5%

while there used to be about 50 calls per day at the start of the pilot, the last week of the pilot has witnessed more than 300 calls every day. The interesting thing to note is that the NGO did not actively train the villagers to use the system nor did they advertise this service aggressively. We had started with a group of about 30 users who were initially contacted to start using the system and provide us feedback. The caller base increased from these 30 users just by word of mouth and has reached even the neighbouring villages. We have had villagers from the neighboring villages calling and creating their advertisements on the VoiKiosk.


Number of calls per day






0 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116

Days since the start of pilot

Fig. 6.

Number of calls per day, starting from the launch of pilot.

While most calls were during the afternoon and late evening times, we were surprised to see calls arriving at 2:00 am. There were a total of 99 calls received between odd hours of 11:00 pm and 5:00 am. Figure 7 shows the calling pattern during different times of the day. This is an aggregate number summed over the number of calls in the four months.


Number of calls per hour




0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

24 Hours of day (starting from 12:00 midnight)

Fig. 7.

Hourly breakup of number of calls in the day.


There were a total of 2528 advertisements recorded by 463 different users. Many people recorded their advertisement multiple times over the period of four months. Although one person can record only one advertisement, people often call again either to improve their advertisement, or to record a fresh advertisement. One caller had recorded his ad 62 times. Though most callers had recorded it only once. The graph in Figure 8 shows the different callers and the number of times they have recorded their advertisements.

behind this was that by listening to the advertisements, a novice user will know what to speak in her advertisement. When the number of advertisements increased beyond 20, we noticed that the users were still patiently listening to them and then would post their advertisement. This usage pattern provides (at least) one of the following two insights about the users: This population is more patient and could listen to a long list of information over phone as compared to what has been observed with the western population’s experience with spoken dialog systems. And/OR The villagers ascertain certain value to posting an advertisement and so are ready to wait for a long time in order to get a chance to post. When the number of advertisements increased beyond about 40, we provided option to jump to the next advertisement by pressing a key on the phone keypad. Perhaps the users were comfortable with navigating the VoiKiosk system by then, and therefore they were able to learn the keypad navigation. It should be noted at this point that during the needs-gathering phase, the users had clearly mentioned that navigation through keypad will not be easy and so a speechinput method should be provided to navigate the VoiKiosk. Even then, the average time spent per-call was higher (104 seconds) after Modification-1, it reduced to 75 seconds when we provided option to record without users having to listen to all advertisements. This is a clear insight that users were able to interact more efficiently with the VoiKiosk system when we provided keypad-based shortcuts for faster navigation. VII. D ISCUSSIONS The advertisement section attracted the most attention, and was accessed the most. Initially, we had a couple of users (a mechanic and a truck owner) upload personal (classifiedtype) advertisements to the VoiKiosk. Inspite of these initial straightforward examples, the villagers soon found some very interesting and innovative uses for this section: • An eighth grade student announced himself and his mobile number. • A man uploaded his profile for matrimonial purposes (to invite marriage proposals). He made several attempts till he was satisfied that all the details he wanted to capture were recorded. – Another person created a ”response advertisement” commenting on the above profile! • A pair of young parents recorded a message in their child’s voice for the child’s grandparents to hear. • A politician posted a thank you note after winning a local election. These examples are very demonstrative of several things: (a) The villagers really understood the technology enough to play with it, and find innovative uses for it, and (b) Even

60 Number of advertisement recordings






0 1 18 35 52 69 86 103 120 137 154 171 188 205 222 239 256 273 290 307 324 341 358 375 392 409 426 443 460 Unique callers

Fig. 8.

Advertisement update frequency of callers.

C. Improvements with Changes in Interface During the four months of the live pilot, we made two changes to the system: 1) Enabled easy navigation of advertisements by providing options to jump to next advertisement through keypad input. 2) Enabled easy recording of advertisement by allowing users to record without them having to listen to any advertisement. The initial system was live for 70 days. After the first change, the system was live for 21 days and then after the second change, it was live for 30 days. Table III shows the number of calls, average call time, number of advertisements and unique callers in the three situations.

It is interesting to note that over a period of these four months, the number of advertisements have been increasing. Even then, due to improved navigation, the time spent by users has not increased significantly. In the first deployment of the system, a caller had to listen to all advertisements before she could post her own advertisement. The reason


though they may have never heard of social networking on the Web, the need for social networking is natural and compelling, and (c) they independently discovered the message/responseto-message interface as well as voice mail (as obvious as these might sound to us). Thus the advertisements section evolved to a message-board by innovative use of the interface by the villagers. As was mentioned in the previous section, in an earlier incarnation of the VoiKiosk, a villager had to listen to all the advertisements before posting his own, but that did not seem to act as a deterrent. The villagers appear to be extremely patient in listening to all the information and spending a lot of time with the system, which may be a reflection of culture or the lack of alternatives, most likely both. Of the 20499 calls received on the VoiKiosk system, only 2532 were for creating the content. Creating the content in the VoiKiosk is either by recording of an advertisement by a villager or by recording of other messages by the Kiosk Operator or the agriculture Expert. Therefore, more than 87% of the calls were for accesing the content on the VioKiosk system while only about 13% calls were made for creating the content. This indicates the usefulness of the content that was being created on the VoiceSite. In the previous section, we have provided more details about the advertisements rather than the agriculture, health and ashwini center. This is because the other three categories were mostly information upload from one specific person (either the Kiosk Operator or the Expert), and the access was to the villagers. Therefore these services, although important, were different from the advertisements because the latter had user generated content. By providing the ability for all villagers to create content, VoiKiosk becomes a much more participatory platform for rural areas. Although the statistical analysis proves the acceptance of the VoiKiosk system by the villagers, the next step is to get usability feedback directly from the users. We plan to talk to a sample of VoiKiosk users and seek their feedback in terms of the usability of the system and the value of the content. We also plan to provide more critical services such as daily crop prices to registered farmers and medication reminders to patients through the VoiKiosk system. It will also be interesting to observe the usage pattern of a different village and identify the similarities and differences and question whether they reflect the demographics of that village. VIII. C ONCLUSION In this paper, we present a information kiosk system for use in rural areas to create and access locally relevant content. We identify the information needs in rural areas by talking to villagers and the NGO. We develop VoiKiosk – a voice-based system that can be accessed by a phone to provide a cost effective solution that is easy to use by the less-literate people in rural areas. We test the initial prototype with the villagers to evolve a participatory design of the final system. The system was then deployed live in a village in South India and we captured user statistics for 4 months. More than 900 users used

the system over a period of four months. Users made about 20000 calls to this VoiKiosk system. We present the usage pattern obtained by different modifications in the VoiKiosk over these four months. We provide a detailed description of how some of the services were used beyond what they were designed for. This leads to an interesting insight into the social networking applicability of the VoiKiosk system. The increasing use of the VoiKiosk system for the different purposes leads us to believe that a voice-based mechanism for local content creation is a very powerful interaction modality to provide information and communication technologies in rural areas. ACKNOWLEDGMENT We thank Byrraju Foundation for supporting VoiKiosk in Andhra Pradesh villages and for providing valuable insights. R EFERENCES
[1] “World Urbanization Prospects: The 2007 Revision Population Database,” http://esa.un.org/unup/.

[2] Ministry of Finance, Government of India, “Economic Survey 2001-2002,” http://indiabudget.nic.in/es200102/chapt2002/chap106.pdf. [3] Population Reference Bureau, “World Population Data Sheet,” no. ISSN 0085-8315, Aug 2006. [4] S. Agarwal, A. Kumar, A. A. Nanavati, and N. Rajput, “Voikiosk: Increasing reachability of kiosks in developing regions,” in Proc. Intl. Conf. on World Wide Web (WWW), China, April 2008. [5] T. S. Parikh and E. D. Lazowska, “Designing an Architecture for Delivering Mobile Information Services to the Rural Developing World,” in Proc. Intl. Conf. on World Wide Web (WWW), May 2006. [6] S. Seshagiri, A. Sagar, and D. Joshi, “Connecting the bottom of the pyramid: An exploratory case study of india’s rural communication environment,” in Proc. Intl. Conf. on World Wide Web (WWW), China, April 2007. [7] B. Kolko, E. Rose, and E. Johnson, “Communication as information-seeking: The case for mobile social software for developing regions,” in Proc. Intl. Conf. on World Wide Web (WWW), Canada, May 2007. [8] D. Ramachandran, M. Kam, J. Chiu, J. Canny, and J. L. Frankel, “Social dynamics of early stage co-design in developing regions,” in Proc. CHI, USA, April-May 2007. [9] C. K. Prahalad, “The Fortune at the Bottom of the Pyramid: Eradicating Poverty Through Profits,” in Wharton School Publishing, 2004. [10] E. Brynjolfsson and L. Hitt., “Paradox lost: Firm-level evidence on the returns to information systems spending.” Management Science, vol. 42, April 1996. [11] “Netcraft Web Server Survey,” http://news.netcraft.com/archives/web server survey.html. [12] K. Matthee, G. Mweemba, A. Pais, G. V. Stam, and M. Rijken, “Bringing internet connectivity to rural zambia using a collaborative approach,” in Proc. Intl. Conf. on Information and Communication Technologies and Development (WWW), India, December 2007. [13] U. Saif, A. L. Chudhary, S. Butt, N. F. Butt, and G. Murtaza, “Internet for the developing world: Offline internet access at modem-speed dialup connections,” in Proc. Intl. Conf. on Information and Communication Technologies and Development (ICTD), India, December 2007.


[14] G. Buchanan, S. Farrant, M. Jones, H. Thimbleby, G. Marsden, and M. Pazzani, “Improving mobile internet usability,” in Proc. Intl. Conf. on World Wide Web (WWW), Hong Kong, May 2001. [15] R. Heeks, “Information Systems and Developing Countries: Failure, Success, and Local Improvisations,” The Information Society, vol. 18, March 2002. [16] E. R. Megwa, “Bridging the Digital Divide: Community Radio’s Potential for Extending Information and Communication Technology Benefits to Poor Rural Communities in South Africa,” Howard Journal of Communications, vol. 18, October 2007. [17] F. Hussain and R. Tongia, “Community radio for development in south asia: A sustainability study,” in Proc. Intl. Conf. on Information and Communication Technologies and Development (WWW), India, December 2007. [18] S. R. Sterling, J. O’Brien, and J. K. Bennett, “Advancement through Interactive Radio,” in Proc. Intl. Conf. on Information and Communication Technologies and Development (WWW), India, December 2007. [19] T. S. Parikh, “Designing an Architecture for Delivering Mobile Information Services to the Rural Developing World,” Ph.D. Dissertation, University of Washington, 2007. [20] T. S. Parikh, P. Javid, S. K., K. Ghosh, and K. Toyama, “Mobile Phones and Paper Documents: Evaluating a New Approach for Capturing Microfinance Data in Rural India,” in Proc. CHI, Canada, April 2006. [21] Y. Schwartzman and T. S. Parikh, “Using CAM-equipped Mobile Phones for Procurement and Quality Control at a Rural Coffee Cooperative,” in MobEA V, Mobile Web in the Developing World, Canada, May 2007. [22] M. Plauche and M. Prabaker, “Tamil market: a spoken dialog system for rural India,” in Proc. CHI, Canada, April 2006.

[23] J. Sherwani, N. Ali, S. Mirza, A. Fatma, Y. Memon, M. Karim, R. Tongia, and R. Rosenfeld, “Healthline: Speech-based access to health information by low-literate users,” in Proc. Intl. Conf. on Information and Communication Technologies and Development (WWW), India, December 2007. [24] R. Veeraraghavan, N. Yasodhar, and K. Toyama, “Warana Unwired: Replacing PCs with Mobile Phones in a Rural Sugarcane Cooperative,” in Proc. Intl. Conf. on Information and Communication Technologies and Development (WWW), India, December 2007. [25] “MobilED Initiative,” http://mobiled.uiah.fi/. [26] T. Leinonen, F. Aucamp, and E. Sari, “Audio Wiki for mobile communities: information system for the rest of Us,” in Workshop on speech in mobile and pervasive environments, Mobile HCI 06, September 2006. [27] A. Kumar, N. Rajput, D. Chakraborty, S. Agarwal, and A. A. Nanavati, “Voiserv: Creation and delivery of converged services through voice for emerging economies,” in WoWMoM’07 Proceedings of the 2007 International Symposium on a World of Wireless, Mobile and Multimedia Networks, Finland, June 2007. [28] S. Agarwal, D. Chakraborty, A. Kumar, A. A. Nanavati, and N. Rajput, “HSTP: Hyperspeech Transfer Protocol,” in ACM Hypertext 2007, UK, September 2007. [29] A. Kumar, N. Rajput, S. Agarwal, D. Chakraborty, and A. A. Nanavati, “Removed for blind review,” in Proceedings of the World Wide Web, April 2008. [30] A. Kumar, N. Rajput, D. Chakraborty, S. Agarwal, and A. Nanavati, “Removed for blind review,” in SIGCOMM Workshop on Networked Systems for Developing Regions, Japan, Nov 2007. [31] “Byrraju Foundation,” http://www.byrrajufoundation.org.


E for Express1: “Seeing” the Indian State through ICTD
Renee Kuriyan and Isha Ray
electricity bill payment and issuance of government certificates and forms. The article drew attention to Andhra Pradesh (AP) in particular as the “hot-bed” of e-governance since it started e-Seva 2 , a project which was upheld as a model of efficient government service delivery [1]. The cartoon accompanying the article, entitled “E for Express” symbolized this new form of service delivery and the accompanying changes in the Indian bureaucracy (Figure 1). It depicted a sleek Indian express train carrying passengers on top of its roof and inside its cars, holding laptops and cell phones. The Indian railways were once emblematic of the country’s bureaucratic and slowmoving ways. At the same time, often associated with Gandhi traveling on their rooftops, the always-overflowing railways were symbols of accessibility to the Indian masses. The passengers in this cartoon, however, wearing expensive business suits and holding laptops, were a far cry from usual images of the “common man.” The cartoon also highlighted the links between modern technologies and an Indian bureaucracy that was changing from being slow and “neither equitable nor efficient” 3 , to one that is fast-moving, capable, and accessible to all. It hinted that with the spread of egovernance services and state of the art technologies the public’s experience of the government was being transformed.

Abstract— This paper examines how, in their attempts to liberalize and modernize their operations, Indian states are using ICTD e-governance services to represent themselves in a new way to their citizens. It reveals how states come to be seen by their citizens through their everyday interactions at ICTD telecenters. The research finds that, with its e-governance services, the state is trying to recast its image to fit marketfriendly principles such as economic efficiency, accountability and effectiveness. Citizens simultaneously trust the government as credible and are disillusioned with it as inefficient. Telecenterprovided e-governance services are partially re-shaping the boundaries between state, civil society and markets. Index Terms—e-governance, partnerships India, state, public-private

I. INTRODUCTION “Paying an electricity bill [in India] could easily involve a day’s wait at a government office where a cross official would demand a bribe for doing his job. The same was true for phone bills, water bills, taxes and all other interactions with government. Often the customer would first have to go to a bank to get a banker’s draft and then take it to a queue at the payment office. Even a small firm would need an employee whose sole task was to pay bills and deal with other aspects of officialdom. Now all of this can be done online.” (The Economist, Special report on technology and government, February 14, 2008) The Economist report cited here, entitled the “Electronic Bureaucrat,” highlighted India as a nation whose government was on the path to technologically “leapfrog the rich world” by putting its services online. It argued that the Indian government was able to serve its citizens more effectively through its pioneering efforts in e-governance, with online
This material is based upon work supported by the National Science Foundation under Grant No. 0326582, the Technology and Infrastructure for Emerging Regions group, Microsoft Research India, and Intel Research. Renee Kuriyan is a Research Scientist with Intel Research, Beaverton, OR, USA (phone: 503 329 0688; e-mail: renee.kuriyan.wittemyer@intel.com). Isha Ray is an Assistant Professor with the University of California, Berkeley, Energy and Resources Group (e-mail: isharay@berkeley.edu).
1 The Economist, Special report on technology and government, February 14 2008

Figure 1: E-governance in India, “E for Express” (Source: Economist, February 14, 2008)

2 E-seva was launched in Hyderabad in 1999 with goals of ”looking at 'service' from the citizens' point of view” and redefining citizen services using state-of-the-art technologies. It is a public private partnership between the Government of AP and private service providers. It provides services such as: payment of utilities bills, government certificates, licenses, permits, transportation department services, bus reservations, passport services, and business services based on agreements with private businesses such as cell phone providers and banks (http://esevaonline.com/). 3 The Economist, Special report on technology and government, 2008


This paper examines the implied hypothesis that, through Information and Communication Technology (ICT) projects, and in particular those that provide e-governance services, the nature and boundaries of the Indian state are being redefined. We do this through an analysis of telecenters that provide egovernance (and other) services in two states in India: Kerala and AP. Both states have been enthusiastic adopters of such telecenters, and both states have formed public-private partnerships (PPPs) in order to promote their e-governance agendas. We ask: Do citizens “see the state” [2] differently in light of governance and development services provided through these entrepreneur-mediated telecenters? The term “see the state” is a twist on the pioneering work of James Scott, who, in Seeing Like a State (1998), examines the ways in which the state sees (and thus controls) its citizens. He argues that the state often attempts to make its populations “legible” through simplified, yet strategic and technical, processes (such as mapping, censuses, and various other standardized modes of representing the population). Conversely, Corbridge et al in Seeing the State (2005) examine how citizens see government agencies, through an ethnographic analysis of state-citizen interactions in eastern India. We build on Corbridge et al to further explore the shifting dynamics of state-citizen relationships through ICTD telecenters in southern India. We begin by highlighting some of the important ideas in the literature on governance, e-governance and their associated reforms in India. Then, using two vantage points -- the view of the state and view of the citizens -- we argue that in both Kerala and AP, the state governments are turning to the private sector as partners not only for the provision of egovernance services, but to associate themselves with a liberal market order and the modernization process more broadly. The state is trying to recast its image to fit market-friendly principles such as economic efficiency, accountability and effectiveness, all of which embody the “good governance” agenda in India and elsewhere. These efforts reflect the state’s attempt to reposition itself in the context of a liberalizing economy and to alter the way in which it is perceived by its citizens. We find that citizens simultaneously trust the government as credible and are disillusioned with it as inefficient. We find that e-governance through decentralized entrepreneur-mediated telecenters are partially redefining perceptions and expectations of the state, the lay citizenry and the private sector. This hybrid version of government is gradually reworking both the way the state sees itself and how citizens see the state. II. METHODS Using a combination of methods such as interviews, participant observation, and literature and document review, we explored the reworking of these relationships through the case of ICT for development (ICTD) telecenters. Primary data collection took place on several trips to India over a period of

3 years from 2004 to 2006. We examined four projects in the states of Kerala and Andhra Pradesh, all of which provided egovernance services to citizens via entrepreneur-mediated telecenters. These were Akshaya in Kerala, and three separate projects in AP called Rural Eseva, Rural Service Delivery Points (RSDP), and Rajiv Internet Village Centers (Rajiv). The telecenter projects offered a glimpse into the realities and perceptions of good governance principles and government policies on ICTD. The telecenters in AP and Kerala focused on e-governance services that 1) provided entitlements (including certificates, licenses), 2) provided information (sectoral, agricultural, or health), 3) provided redress for grievances, and 4) enabled government bill payments (electricity, taxes, utilities). In Kerala the telecenter project offered computer education services as well. We conducted 31 interviews with state actors within the Government of India (GOI) and within the AP and Kerala state governments, using a semi-structured interview protocol. These interviews explored each state’s strategies for delivering e-governance services, and how states perceived their roles and those of the private sector in the process. We conducted open ended, key-informant interviews with 16 local entrepreneurs in Kerala and AP from the four different projects. Through these interviews and shorter conversations with other entrepreneurs, we explored the role of entrepreneurs in delivering governance services and the position of the entrepreneur with respect to the state. In the telecenters of each project, we observed entrepreneurial behavior, and engaged in informal conversations with the users. We interviewed telecenter users and non-users in each district, using an open-ended interview protocol, exploring perceptions of both the project and the state with respect to their roles in development. Sixty-five interviews with household members were conducted in Kerala, and 70 in AP. All interviews were recorded, transcribed and coded using interview analysis software. Finally, we analyzed literature and policy documents on good governance, ICTD, and the modern Indian state. III. GOVERNANCE REFORMS AND E-GOVERNANCE IN INDIA The expression “governance” is ambiguous and has become a catch-all term in the study and practice of development. The academic literature on governance is varied, containing perspectives from institutional economics [3], sociology [4, 5], development studies [6] and governmentality-oriented theory [7]. From a purely technocratic perspective, governance can be understood as a set of administrative or managerial tools, which, properly applied, lead to “good governance.” Administrative reforms of Western governments in the 1980s and 1990s, implemented under the marketoriented framework of new public management (NPM,[8, 9] underscored concepts such as efficiency, open markets, accountability, customer service and decentralization. NPM models indicated that market-based mechanisms for service delivery were more competitive and therefore more efficient than traditional government-based provision. The overall philosophy was that governments should be encouraged to perform like incentive-driven private businesses and


entrepreneurs. Similar principles were later invoked in “good governance” reform programs in developing countries. Development practitioners in recent years have defined good governance principles as transparent policymaking, professionalism and accountability in government actions, and civil society participation. International development agencies have sought to operationalize good governance by restructuring and privatizing state bureaucracies, decentralizing state power through local government or nonstate actors, reforming legal systems, and implementing public private partnerships [10]. The rationale behind these reforms was that by “combating corruption, nepotism, bureaucracy, and mismanagement … aid would be effectively used to achieve the objective of reducing poverty” [11: 270]. Critics of good governance argued that it had a neoliberal bias towards market-led development due to the influence of donors. Critics also charged that the words ‘governance’ and “government” were being used interchangeably because: “Governance appears to be used in place of government as if “government” was a difficult word to sell in a privatized, market-oriented society. Governance is about a ‘reinvented’ form of government, which is better managed” [12: 18]. In India, the good governance agenda developed against the background of a long history of concern over state-society relationships rooting back to India’s anti-colonial struggles [2]. Debates about good government with respect to appropriate power for India’s citizens, corruption, and abuse of executive powers have been part of the national agenda from the 1960s. The reforms of the 1990s, which liberalized the economy to a great extent, were introduced largely to address India’s balance of payments problem [2], But a broader program of administrative and economic reforms was also supposed to tackle political challenges, or what Kohli called India’s “growing crisis of governability” [13: 23]. By this time, the centralized state had lost a great deal of legitimacy, and devolution through administrative reforms promised a range of benefits [14]. Prime Ministers from the 1990s onwards turned to a mix of deregulation, privatization, civil service reform, decentralization and PPPs to address a range of administrative “failures” in India. Decentralization in particular was an integral part of the governance reforms and was viewed as a way to reduce the role of the state by fragmenting its authority and making it more responsive and efficient. Local governments were supposed to have better information on local needs, and were considered closer to their constituencies and thus more politically accountable, than centralized government agencies. Decentralization would thus “expand service deliveries as authority goes to those more responsive to user needs”[15: 173]. However, even local government institutions could lack accountability mechanisms and be vulnerable to corruption or to being captured by local elites. This frequently frustrated the goals of equitable public provision of services to the general population (Ibid). In addition to decentralization, therefore, the administrative reforms included PPPs in service delivery. Since Indian states were often in fiscal trouble, with limited budgets to deliver government services, they also needed private sector partners

for help with financial contributions and the modernization process overall. During our research, several government officials talked about the financial constraints of the government and its inability to scale service delivery without private sector participation. For example: “The government is not having many funds. The private sector has funds and if the government supports [them] they can invest. In Kerala, [the] government is running out of money. We go for heavy loans from the Asian Development Bank... For the last ten years it’s been like that… There is a lot of work [to be done], but not much money.” (Interview, IT Mission, Kerala) Officials in Kerala indicated that inadequate technical capacity was another reason why the government needed to move towards a PPP model for service delivery: “There was no capacity with the government. That had to be built up and that is not something you can do overnight. In e-governance you find a much higher acceptance of the private sector as a player. The government acknowledges that they don’t have the technical or financial capacity in the government and they need to look to the private sector” These constraints and incentives led several Indian states to introduce decentralized e-governance delivery projects through PPPs. E-governance measures aimed to improve administrative processes by using ICTs, and to build connections to promote socioeconomic activity [16]. In accordance with the principles of good governance, egovernance was meant to emulate the private sector’s qualities of reliability, transparency, scalability and treating citizens more like customers. The mainstream press was generally supportive of the promise of fewer lines, administrative efficiency and the electronic bureaucrat (Figure 2). For example, the Indian Express newspaper in a 2005 article credited e-governance with bringing citizens to the center of service delivery: “E-governance has, in many cases, restored the choice to the citizen as to the quality and adequacy of services he is entitled to expect from public organizations. Citizen-centric governance meant government was for the people and the services were tailored to meet their requirements. 4 ”

44 “Giving to people their new right: The right to good governance” (http://www.indianexpress.com/india-news/full_story.php?content_id=76910)


Figure 2: The Electronic Bureaucrat (Source: Economist, February 14, 2008)

The focus on e-governance led many state governments to adopt “integrated citizen service portals” or ICTD telecenters. Telecenters (or kiosks) were meant to bring government and other services closer to citizens and businesses through partnerships with local entrepreneurs [17]. Telecenter entrepreneurs had market-based incentives to perform efficiently; were situated at a village level; and in most cases were members of the communities that they served. Consequently, it was felt, they were best suited to make egovernance services responsive to the felt needs of the local population. Despite the Indian states’ embrace of e-governance through telecenters, there have been a number of critiques of these projects. It has been argued that telecenter services were not reaching the poor [18], were unsustainable [19, 20], and faced political challenges in actually delivering e-governance services [5]. We now examine the influence of this new form of governance from the perspectives of the state and of the citizens that these initiatives are meant to serve. IV. CONSTRUCTING THE STATE AND CHANGING ITS IMAGES Our research aims to understand how the state represents itself to its citizens and how it comes to be seen by them through their everyday interactions at ICT telecenters. Several works of political anthropology depart from the conventional treatment of the state as an institution defined by its powers and operations, and instead examine the state as it is ‘imagined’, constructed and conceptualized through the experiences of its citizens [21-24]. These works analyze how the state becomes “socially effective through particular imaginative and symbolic devices”[25: 981]. Gupta (1995) argues that through ethnographic research on the state, for example with lower level officials and politicians, it is possible to “illuminate the quotidian practices of bureaucrats that tell us about the effects of the state” on the lives of ordinary people [24: 376]. What quotidian practices “construct” the state? Corbridge et al. (2005) argue that state-citizen interactions are based on

everyday flows of power, money, commodities and information. The poor in rural India form their understandings of the state through financial and legal transactions at local government offices [2]. They note that a poor person most directly experiences the state when he or she registers for birth or death certificates, receives a registration form, or picks up an entitlement. These are many of the e-governance services now offered by entrepreneurs in telecenters. With telecenter entrepreneurs in effect replacing the functions of local bureaucrats, we examined the ways in which the state is being “seen” or experienced by different groups of people accessing e-governance services through ICT telecenters. Our research indicates that at least some segments of the rural and periurban population have developed new images of the state through their encounters with telecenter entrepreneurs. They evaluate these experiences against their previous and often negative encounters with the local arms of the state. We argue that both Kerala and AP, through changing their governance initiatives, are deliberately re-working images of the state versus the private sector. The growing acceptance of service delivery through ICTs, the political reality of economic liberalization policies, and discourses of India as a technological leader in e-governance have all influenced the types of images the state tries to portray to its citizens. We find that how the state is then perceived by civil society depends on a combination of factors -- the ambience of telecenters compared to government offices; how entrepreneurs interact with the public; how the state brands the telecenters to citizens (whether it is considered a private or a government office); and the extent to which citizens both trust, and are disillusioned with, the government. Our observations in Kerala and AP support neither the strong proponents of the good governance agenda who advocate a minimized role for the state, or the strong opponents who fear that service through PPPs will, in effect, ‘privatize’ the state. We find that, rather than removing the state and supplanting it with the private sector, telecenter projects under the PPP model have created a space for the state to construct a better image of itself with respect to its citizens. These telecenters and their electronic delivery of services are being used as, and to some extent are becoming, symbols of responsiveness and of accessibility to all. The partnering of the state with private entrepreneurs is a key mechanism through which the state is trying to recast its image and be re-imagined by its citizens. A. Representations of the State in Kerala and AP Throughout India, at e-governance and telecenter conferences, and in our discussions with state officials at the central government and AP and Kerala state government levels, we found a similar discourse about the state’s need to change the way it treats, and how it was perceived by, citizens. There was a general consensus among government officials that the state needed to behave more like the private sector in service delivery. Officials in Kerala and AP indicated that one way to accomplish the change was to partner with private entrepreneurs who had an economic incentive to provide good customer service.


Entrepreneur-run telecenters represented a new concept of government service delivery that was fast, efficient, hassle free, and accessible to the “common man” (Figure 3). These attributes allowed states to renegotiate their standing with their citizens, a standing that was clearly low on account of previous encounters between the state and its citizens. A former AP official was clear on the expected differences: “The appearance is totally unlike a government office. The ambience created is more of a private sector feelingit has a uniform ambience built across all the service centers across the state- same design, same color. You have a feeling that you have not walked into a government office, but you’ve walked into something the private service sector would run. The beauty is that it starts at 8 in the morning and closes at 8 at night.”

Figure 3: Old versus new images associated with government offices and services. (Photographs shown in presentation at National e-governance conference in Kochi (2006) to show old and new forms of government service delivery)

Our conclusion that both AP and Kerala were quite concerned about their public images is consistent with Madon’s (2005) study of image formation in the state of Kerala. She found that, given the historical resistance to and suspicion of the private sector in Kerala, the government was careful to manage the image of the Akshaya project. Rather than advertising the Akshaya project to the public as a PPP at first, it initially portrayed the project as social development bringing e-literacy to all. After a few years the government began to emphasize the private sector’s role in delivering services to citizens through state-entrepreneur partnerships. Madon attributed this change to the growing optimism and acceptance of IT in governance reforms and increasing confidence in the private sector in Kerala. The Kerala government in this case judiciously managed the image of the role of public-private partnerships and governance reform. Despite their acceptance of private sector participation, state officials in Kerala and AP asserted that the credibility of the government brand was highly important for the PPP-based implementation of government services. One official we interviewed in AP stated, “The image is that these private telecenters are the government. If you look at the transactions- there are millions a month. Probably 99% are government to citizen transactions… they are branded as government. That is why the credibility is much better. Because people will think twice if somebody else [who is private] wants to collect your electricity bill and pay taxes. People will think twice!” (Interview, 2006). State officials asserted that despite its reputation for poor quality and slow services, some level of government association with the entrepreneurs created credibility for the telecenters. The images of the state were constantly being constructed and renegotiated not only by state and project officials, but by citizens themselves, as we show in the next section. B. “Seeing the state” through entrepreneurs Citizens “see the state” through the individuals who represent it. Because the personal characteristics of key individuals, such as a pro-poor officer or a corrupt official, often appear large in the minds of citizens who have experienced them, the views of government are likely to be fractured [2] The local bureaucrat has traditionally embodied the “state” in India and the state is then “seen” through the everyday interactions between ordinary people and this person-cum-state: “The manner in which these officials negotiate the tensions inherent in their location in their daily practices both helps to create certain representations of the state and powerfully shape assessments of it, thereby affecting its legitimacy.” [24:388] Corbridge et al argue that “encounters with the developmental state build up a dynamic picture of “it”, both as an idealized set of values and practices (the state as it should work) and also as its flawed but more commonly experienced

Frequent reference to the telecenters’ appearance or ambience revealed that state employees were at least as concerned with their image as they were with making government services more streamlined. Several officials acknowledged that government offices had historically been very different in appearance than something that the private sector would run. They recognized that, unlike government offices in which officials often closed early, were not present at their offices, and were on the whole unaccountable, these new telecenters had much more responsiveness and flexibility. State employees explained that with telecenters, they were aiming for transparency, speed, and convenience for the citizen, and that wanted the government to be seen as more “common man friendly”. Our research showed that the business-like appearance of the telecenter actually represented a type of streamlining to users and also to the officials. Both states emphasized the business-like aspect of these telecenter projects, a feature that is at the core of the good governance agenda. State actors, especially in AP, insisted that the credibility of the telecenter and e-governance projects could only be achieved when they were managed by the private sector because of their good services, longer hours, lack of resource constraints and flexibility. But both state governments wanted to make sure that the government’s name and brand was associated with these “private” telecenters. By having the private sector deliver the services but branding the centers with the government name, the state had found a concrete way to represent itself as protecting the public good and doing so in a business-like manner.


counterpart (the state as it does work)” (2005: 119). Today, the local bureaucrat is no longer the exclusive embodiment of the state. When accessing ICT telecenters, citizens’ everyday interactions with the state are now being mediated by smallscale entrepreneurs as they, and not local officials, provide a host of government services to the populace 5 . These entrepreneurs, even though they are not state employees, come to represent the state at least in part, and also affect its legitimacy. During our research many entrepreneurs emphasized their ability to deliver government services through the PPP strategy better than the government could deliver them. However, they relied on being “seen” as the government, particularly when they collected electricity bill payments or issued government certificates. Entrepreneurs confirmed what state officials had claimed, that being thought of as the government or having some government endorsement gives people the confidence to pay their bills at the telecenters. The association with the government also prevented people from thinking the telecenters were corrupt. An entrepreneur in AP admitted: “If people think we are working under private people they won’t come. I think and tell villagers that we got a loan from the government to establish this center. So it is a government center. For the most part, entrepreneurs providing e-government services to citizens benefited from the association with the government’s name. Entrepreneurs faced challenges, however, branding their centers as “government” institutions when they provided more than just e-governance services to citizens, such as computer education courses[19]. Rajalekshmi’s study of the Akshaya project also found that the most important reason for its acceptance was “the trust that people had in government as an institution and the fact that this project was spearheaded by the government” [17: 29]. Our study in both AP and Kerala corroborated this observation; citizens’ trust in government institutions allowed these privately-run payment and other e-governance systems to function. C. Citizens’ perspectives We now turn to the perspectives of the households. Several of our respondents said that the private sector, meaning the telecenter entrepreneurs, treated all customers with a level of respect that they did not receive from the government. Our interviews with households revealed that ordinary citizens, especially the poorer ones, resented the bureaucratic and rude manner in which government officials often treated them. One woman, living below the poverty line, complained:
5 Undoubtedly, the range of people’s encounters with the state extends much beyond their interactions with telecenter entrepreneurs. Nor do we wish to imply that telecenter entrepreneurs are the only intermediaries between the state and its citizens: field-level government employees and NGO representatives, for example, also play this role. But telecenter entrepreneurs are a novel form of intermediary in that they are simultaneously the state and not the state, at least when they are providing government sanctioned egovernance services.

“At government offices, people are not helping me. Even when we go to a government office, they won’t help us people like us. Even if you have a small job, they will not give any amenities, like ration card. They will have to give, but they won’t give it. We have to do everything on our own. Here [at the telecenter] there is more respect for people. “ While poorer citizens seemed concern with the need for more respect from the government, middle-class individuals complained that traditional service delivery methods were slow and inefficient. A middle-income elderly man described going to a government office compared to a private telecenter to pay his electricity bills: “At a Government office, a person has to wait 5-6 hours. A person uses Rural e-seva now. Before they had to travel for many hours to many offices. [Now] instead of 4 hours, I use 10 minutes. We use [Rural] e-seva for electricity bills and telephone.” Speed and convenience were not the only advantages offered by the telecenters. Household members seemed to think that dealing with private providers was easier overall compared to dealing with employees at a government office. In the words of a middle aged man: “It is easier to deal with these private people than the government - if you have 2-3 private centers, definitely the private person will grab people by providing offers, private institutions, providing amenities. They give you water. You won’t find water in a government office. They [the private centers] provide good amenities to you.” In this sense, it seemed that the government had outsourced the provision not only of services but also of customer satisfaction to the private sector. Thus we found that citizens had mixed feelings towards the government with respect to basic governance services. Ordinary citizens trusted the credibility of the government name but were dissatisfied with the quality of government services, and with having to put up with rude and bureaucratic treatment in an often-corrupt system. Citizens had similarly divided feelings towards the private sector. Civil society in India may be disillusioned with government provided services, but it was equally skeptical of the private sector as the protector of the poor. Hansen’s study of the Indian state as a guarantor of order stated that, “It may be well that ordinary Indians are less in awe of the state than a few decades ago, but it is still regarded as indispensable for public order and for recognizing communities, leaders or claims as legitimate.”[26: 37]. The resistance to private sector participation in service delivery among some segments of civil society was rooted in the belief that this sector was solely concerned with profit making and was frequently corrupt [27]. Thus, just as the state officials and entrepreneurs indicated, the citizens we interviewed agreed that, despite a reputation for poor services, the government brand was nevertheless accountable and credible, particularly for the delivery of governance services. These


divided feelings explain why the state sees the private entrepreneurs as indispensable to its new image and why the entrepreneurs see the state as equally indispensable to their ability to provide e-governance services. D. The hybrid state We found that through these telecenter projects a new form of government is being experimented with that attempts to combine the accountability of an elected government with the efficiency and customer service associated with the private sector. The state’s new vision of itself contrasts sharply with the images that its population might have had from previous encounters at conventional state offices. This hybrid version of government is gradually reworking both the way in which citizens with access to ICTD telecenters now perceive it and also the state’s perception of itself in relation to its citizens. Scholars of the Indian state have argued that, at the most local levels, the boundaries between state and civil society are often blurred [24]. Through neighborhood ICT telecenters, the lines between states and civil society appear to merge as telecenter entrepreneurs hold multiple positions as local community members, private actors, and also representatives of the state in the delivery of government services. By negotiating their multiple roles in their daily practices, entrepreneurs create representations of the state that affect both its legitimacy and their own. The running together of the public and the private realms is especially evident where profit-seeking entrepreneurs offer public services in a market setting. When walking into an ICT telecenter, citizens encounter painted signboards that mark the telecenters as both private and public. In the case of the Rajiv Internet centers, the signboards might advertise the brands of more than 7 different public and private entities (See Figure 4). The state’s name and authority still define the terrain on which private entrepreneurs have to operate. At the same time, entrepreneurs create new experiences that alter the way in which the state is seen by citizens. Everyday interactions with telecenter entrepreneurs contribute to citizens’ sense of how, and for whom, government operates. Rather than thinking of the state and market as distinct spheres, our research found that, in these part-public part-private telecenters, the very definitions of the state and the market were mutually constituted.

Figure 4: Signboard outside Rajiv Internet Village Center (2006)

E. Conclusion This paper analyzed how, in their attempts to liberalize and modernize their operations, Indian states are using ICTD to represent themselves in a new way to their citizens. Equally, it reveals how modernizing states come to be seen by the citizens through their everyday interactions at ICTD telecenters. We find that e-governance initiatives in Kerala and Andhra Pradesh are not only about delivering services efficiently (“e for express”) but also about updating the state’s image to one that is modern and market-friendly. This goal has partly been achieved via the delivery of government services through telecenters, implemented in partnership with local entrepreneurs. We find that citizens with access to these telecenters generally prefer them to conventional government offices. In particular, the poorer citizens feel they are treated with some respect at the centers, while middle class citizens appreciate the conveniences and amenities that these new centers offer. As the good governance agenda has taken hold in India, Kerala and particularly Andhra Pradesh have embraced the language and the ideal of the modern state conducting its business in a business-like manner. This embrace has been only partial, however. With service delivery via privatelymanaged telecenters, we do not find that the governments in Kerala or AP have been privatized or have withdrawn, as the critics of the “good governance” agenda frequently aver (or as the proponents of the agenda tend to promise). Rather, they play a critical role in managing and constructing their image to their citizens. Citizens’ simultaneous trust in and disillusionment with their governments, combined with their simultaneous admiration for and suspicion of private enterprise, have kept the states practically and discursively important to the success of e-governance. Given that changes in perceptions of the state (for the most part in a positive way) is one outcome of these projects, our research suggests that policymakers could explore how, and if, these changed perceptions can be used in evaluating projects. Given the widespread implementation of telecenters throughout India and the government’s commitment to implement 100,000 telecenters in the country, our research also raises the question: Will citizens in other parts of India encounter the state via telecenters in a similar way as the citizens of AP and Kerala? We note that, as of now, most Indians still do not have access to e-governance, and most government business is still not conducted electronically. We agree with those who argue that the transparency and efficiency expected from public-private partnerships do not always materialize, and with critics of the rush towards telecenters when the results of such projects have been mixed. But our research does indicate that, in Kerala and AP, these public-private telecenters have created a space for the state to renegotiate its role and image in public service delivery.

[24] [25] [26] [27] A. Gupta, "Blurred Boundaries: The Discourse of Corruption, the Culture of Politics, and the Imagined State," American Ethnologist, vol. 22, pp. 375-402, 1995. J. Ferguson and A. Gupta, "Spatializing states: towards an ethnography of neoliberal governmentality," American Ethnologist, vol. 29, pp. 981-1002, 2002. T. Hansen, "Governance and myths of state in Mumbai," in The Everyday State and Society in Modern India, C. J. Fuller and V. Benei, Eds. New Delhi: Social Science Press, 2000. R. Vyas, P. Small, and K. De Riemer, "The private public divide:impact of conflicting perceptions between the private and public health care sectors in India," Int Journal Tuberc lung Dis, vol. 7, pp. 543-549, 2003.

E. Lucas, "The Electronic Bureaucrat A Special Report on Technology and Government," The Economist, vol. February 16,2008, 2008. [2] S. Corbridge, R. Wiliams, M. Srivastava, and R. Veron, Seeing the State: Governance and Governmentality in India. Cambridge: Cambridge University Press, 2005. [3] O. Williamson, "The New Institutional Economics: Taking Stock, Looking Ahead," Journal of Economic Literature, vol. 38, pp. 595613, 2000. [4] J. Kooiman, Governing as governance. London: Sage Publications, 2003. [5] S. Madon, "Governance lessons from the telecenters in Kerala," European Journal of Information Systems, vol. 14, pp. 401-416, 2005. [6] R. Jenkins, "The emergence of the governance agenda: sovereignty, neoliberal bias, and the politics of international development," in The Companion to Development Studies, V. Desai and R. Potter, Eds. NY: Oxford University Press, 2002. [7] N. Rose, Powers of Freedom: Reframing Political Thought. Cambridge: Cambridge University Press, 1999. [8] A. Cordello, "E-government: towards the e-bureaucratic form," Journal of Information Technology, vol. 22, 2007. [9] L. Kaboolian, "The New Public Management: Challenging the Boundaries of the Management vs. Administration Debate," Public Administration Review, vol. 58, pp. 189-193, 1998. [10] S. Batterbury and J. Fernando, "Rescaling Governance and the Impacts of Political and Environmental Decentralization: An Introduction," World Development, vol. 34, pp. 1851-1863, 2006. [11] V. Nanda, "The Good Governance Concept Revisited," ANNALS, AAPSS, vol. 603 January 2006, pp. 269-283, 2006. [12] G. Stokker, "Governance as theory: five propositions," International Social Science Journal, vol. 50, pp. 17-28, 1998. [13] A. Kohli, Democracy and Discontent: India's Growing Crisis of Governability. Cambridge: Cambridge University Press, 1990. [14] P. Bardhan, "Decentralization of Governance and Development," The Journal of Economic Perspectives, vol. 16, pp. 185-205, 2002. [15] P. Bardhan, "Governance issues in delivery of public services," Journal of African Economies, vol. 13, pp. 167-182, 2004. [16] R. Heeks, "Understanding E-governance for Development," Institute for Development Policy and Management, vol. Working Paper No.11, 2001. [17] K. Rajalekshmi, "E-governance services through telecenters: The role of human intermediary and issues of trust," Information Technologies and International Development, vol. 4, pp. 19-35, 2008. [18] J. Thomas and G. Parayil, "Bridging the Social and Digital Divides in Andhra Pradesh and Kerala: A Capabilities Approach," Development and Change, vol. 39, pp. 409-435, 2008. [19] R. Kuriyan, I. Ray, and K. Toyama, "Information and Communication Technologies for Development: The Bottom of the Pyramid Model in Practice," The Information Society, vol. 24, pp. 93-104, 2008. [20] J. Pal, S. Nedevschi, R. Patra, and E. Brewer, "A MultiDisciplinary Approach to Studying Internet Kiosk Initiatives: The Case of Akshaya," Proceedings of the Global e-development Conference, 2004. [21] J. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition have Failed. New Haven: Yale Agrarian Studies Series, 1998. [22] G. Joseph and D. Nugent, Everyday Forms of State Formation: Revolution and the Negotiation of Rule in Modern Mexico. Durham: Duke University Press, 1994. [23] J. Comaroff, "Reflections on the Colonial State in South Africa and Elsewhere: Factions, Fragments, Facts, and Fictions," Social Identities, vol. 4, pp. 321-361, 1998. [1]


Evaluating the Accuracy of Data Collection on Mobile Phones: A Study of Forms, SMS, and Voice
Somani Patnaik, Emma Brunskill and William Thies
In migrating mobile data collection from PDAs to cell phones, a critical issue is ensuring the accuracy of data entry. In the context of healthcare, an errant entry may prevent lifesaving treatments from reaching patients, or may cause the prescription of unnecessary treatment that is costly and dangerous. In financial applications, entry errors may jeopardize the economic standing of communities that are already very poor. Due to the importance of this issue, several researchers have studied the error rates incurred as PDAs are deployed in developing regions. As detailed later (in Table II), the error rates are generally less than 2% (i.e., 2 errors per 100 entries) in programs where users received at least an hour of training [12], [24], [4]. However, in the context of mobile phones, studies of data accuracy are distinctly lacking. The closest work is by Parikh et al., where a hybrid system of paper forms and camera-equipped mobile phones has demonstrated error rates of less than 1% [28]. For standalone data collection on low-end phones, we are unaware of any previous study with a rigorous evaluation of data accuracy. This research opportunity is highlighted in Table I. In this paper, we provide a quantitative evaluation of data entry accuracy using low-cost mobile phones in a resourceconstrained environment. We evaluate three practical user interfaces for entering data on a mobile phone: electronic forms, SMS, and voice. Electronic forms consist of numeric fields and multiple-choice menus, and can be implemented in Java or a native phone platform. The SMS interface requires users to send a structured SMS messages to a server, with logical fields separated by delimiters in the message. The voice interface represents a normal telephone call, with a live human operator that enters the data into a centralized spreadsheet. We evaluated these interfaces in a study of 13 health workers and paramedical staff over a month-long period in Gujarat, India. Each participant was trained and evaluated on all of the interfaces. We focus on the collection of health data relevant to tuberculosis (TB), as we anticipate deploying an electronic system in a real TB treatment program. The data in this paper represent only simulated patient interactions. Our results indicate an error rate of 4.2% for electonic forms, 4.5% for SMS, and 0.45% for voice. These represent the fraction of questions that were answered incorrectly; as each patient interaction consisted of eleven questions, the probability of error somewhere in a patient report is much higher. For both electronic forms and SMS, 10 out of 26 reports (38%) contained an error; for voice, only 1 out of 20 reports (5%) contained an error (which was due to operator transcription). As detailed in Section VI, error rates
Abstract—While mobile phones have found broad application in reporting health, financial, and environmental data, there has been little study of the possible errors incurred during mobile data collection. This paper provides the first (to our knowledge) quantitative evaluation of data entry accuracy on mobile phones in a resource-poor setting. Via a study of 13 users in Gujarat, India, we evaluated three user interfaces: 1) electronic forms, containing numeric fields and multiple-choice menus, 2) SMS, where users enter delimited text messages according to printed cue cards, and 3) voice, where users call an operator and dictate the data in real-time. Our results indicate error rates (per datum entered) of 4.2% for electronic forms, 4.8% for SMS, and 0.45% for voice. These results caused us to migrate our own initiative (a tuberculosis treatment program in rural India) from electronic forms to voice, in order to avoid errors on critical health data. While our study has some limitations, including varied backgrounds and training of participants, it suggests that some care is needed in deploying electronic interfaces in resource-poor settings. Further, it raises the possibility of using voice as a low-tech, high-accuracy, and cost-effective interface for mobile data collection.

I. I NTRODUCTION Mobile devices have shown great promise for improving the efficiency and effectiveness of data collection in resource-poor environments. Compared to a traditional process that relies on paper-and-pencil forms with subsequent transcription to a computer system, mobile devices offer immediate digitization of collected data at the point of survey. This allows for fast and automated data aggregation. It also improves adherence to complex or context-dependent questionnaires, as the device determines which questions should be answered or skipped. The benefits of mobile data collection have been demonstrated mostly in the context of personal digital assistants (or PDAs) [31], [10], [8], [2], [32], [12], [24], [9], [16], [4], [3], [15]. Given the recent explosion of mobile phones around the world, there is growing excitement in extending the successes achieved on PDAs to a phone-based platform. While high-end phones provide the same capabilities as PDAs, low-end phones lack features such as high-resolution displays and touch-screen capabilities. To empower the full population of nearly 4 billion mobile phone subscribers [26] with the capabilities of mobile data reporting, it will be important to establish usable interfaces that are portable to inexpensive phones, and there have been a number of recent efforts in this space (see for example [13], [1], [22], [25], [7]).
Manuscript received September 22, 2008. Somani Patnaik and Emma Brunskill are with the Massachusetts Institute of Technology (email: somanip@mit.edu, emma@csail.mit.edu). During this research, William Thies was affiliated with both the Massachusetts Institute of Technology and Microsoft Research India (email: thies@microsoft.com).

PDAs Published error rates Malaria monitoring in Gambia [12] Clinical study in Gabon [24] Tuberculosis records in Peru [4] Sexual behavior surveys in Peru [3] SATELLIFE [15] DataDyne EpiSurveyor [31] EpiHandy [10] Infant health in Tanzania [32] e-IMCI project in Tanzania [8] Respiratory health in Kenya [9] Tobacco survey in India [16] Ca:sh project in India [2] Cell Phones


Other programs

Cell-Life in South Africa [13] Jiva TeleDoc in India [1, p.42] Pesinet in Mali [22] Malaria monitoring in Kenya [25] Voxiva Cell-PREVEN in Peru [7]


are distinctly higher for health workers than for hospital staff, though this difference may also be influenced by variations in our training environment. We were surprised and alarmed by these results. In our own treatment program, our original intent was to utilize electronic forms. However, we consider it to be an unacceptable risk that 38% of submitted forms – containing critical health information – may contain errors. For this reason, we have overhauled our plans and will implement a treatment program using voice rather than forms or SMS. While the cost of a live operator may be prohibitive in many countries, in India it proves to be very cost-effective. The increased cost of a human operator is more than compensated by the decreased cost of voice-only handsets, voice-only cellular plans, decreased training time, and decreased literacy requirements for health workers. We offer a more detailed analysis in Section VII. While the results of this study have changed our own approach to implementing mobile data collection, we caution the reader in extending the results of the study beyond its original context. In particular, we are focused on the scenario in which users have limited cell phone familiarity and there is limited time to perform training. If either of these variables changes, it may be possible to implement high-accuracy mobile data collection with electronic forms or SMS. Also, while the error rates that we report on mobile phones are 3-8x higher than those previously reported for PDAs, our data are unable to distinguish whether this difference is due to the devices, or due to other aspects of the study demographics, training, and evaluation. A future study could address this question directly by evaluating both phones and PDAs in the same context. Despite these limitations, our study is the first (to our knowledge) that evaluates data entry accuracy on mobile phones. Based on our results, we submit only that electronic forms and SMS may need further validation before gaining widespread deployment in accuracy-critical applications, and that voice may deserve more attention as a high-accuracy and low-cost means of data collection. The rest of this paper is organized as follows. We start by reviewing related work on mobile data collection (Section II). Then we consider the tradeoffs between electronic forms, SMS, and voice (Section III) and detail our implementation

of each interface (Section IV). We describe the setup of our user study (Section V) and the results obtained (Section VI), and we discuss the implications (Section VII). We conclude in Section VIII. II. R ELATED W ORK As summarized in Table I, there have been several initiatives to apply PDAs and cell phones for mobile data collection in the developing world. While a fraction of the studies on PDAs includes an experimental analysis of the error rate incurred, we are unaware of any study which systematically measures the accuracy of data entry on a cell phone. This is the principal novelty of our work. Lane et al. provides a review of nine randomized controlled trials that compare the effectiveness of PDAs and paper forms for data collection [21]. Six of the trials reported entry accuracy, with varying results: two studies found PDAs to be more accurate than paper [20], [29], three studies found the accuracy to be similar with both methods [17], [23], [36], and one study found that paper was more accurate [35]. None of the trials were in the context of the developing world (they took place in North America and Europe). Previous studies of PDA entry accuracy in the developing world are summarized in Table II. In cases where workers received at least an hour of training, error rates are under 2% (i.e., 2 errors per 100 questions). As early as 1991, Forster et. al evaluated the use of PDAs for a malaria morbidity study in the Gambia [12]. Employing secondary-educated workers who received five days of training, they report error rates between 0.1-0.6% and argue that the PDAs offer improved accuracy and efficiency over paper forms. Missinou et al. employed PDAs in a clinical study in Gabon, employing four clinicians who had no prior PDA experience and received 8 hours of training [24]. They report a 1.7% rate of discrepancy between PDAs and paper forms, and note that clinicians preferred the PDAs. Blaya et al. found that error rates improved from 1.3% (with paper forms) to 0.37% (with PDAs) in reporting tuberculosis bacteriology data in Peru1 [4]. The authors also argue that PDAs are cost-effective [5].
1 Blaya et al. reports errors per form, rather than errors per entry [4]. Via personal communication with the author, we determined that there were an average of 7.5 entries per form, yielding the error rates quoted here.

Application Malaria morbidity [12] Clinical study [24] Bacteriology data [4] Sexual behavior [3] Location Gambia Gabon Peru Peru PDA Psion Organizer II XP Palm m500 Palm Zire Palm Zire Education Level Secondary 3 M.D.s, one clinical officer Post-secondary (2-3 years) Secondary or less Training 5 days 8 hours 16 hours 2-3 mins Error Rate 0.1%-0.6% 1.7% 0.37% 14%


Higher error rates have been reported in the case of self-administered surveys, when limited training is possible. Bernabe-Ortiz et al. evaluate the use of PDAs for surveys of sexual behavior in Peru [3]. To protect patient privacy, the PDAs were intended for use by actual subjects, rather than by health workers. As only some subjects had finished secondary education, and subjects received only 2-3 minutes of training, the authors observed a 14% discrepancy between electronic and paper forms. However, the error rate was substantially lower for subjects who had finished secondary schooling. Additional programs have applied PDAs for data collection in the developing world, but have not provided a rigorous analysis of entry accuracy. SATELLIFE uses PDAs for disseminating and collecting medical information in numerous countries [15]. There are anecdotal reports that the PDAs improved data quality [19], and the benefits of decreased error rates were estimated on a five-point scale [6]. Users of the system have also rated its usability [11]. However, we are unaware of a quantitative assessment of the error rates incurred. DataDyne EpiSurveyor [31] has been widely deployed for data collection in Sub-Saharan Africa; while it has been argued that the system is more accurate than paper forms [30], we are unaware of a controlled study. EpiHandy also provides tools for deploying electronic forms on PDAs and has been deployed in South Africa, Uganda, and elsewhere [10]. PDAs have also found application for gathering infant mortality data in Tanzania [32], for pediatric care (as part of the e-IMCI project) in Tanzania [8], for assessing respiratory health in Kenya [9], for surveying tobacco use in India [16] and for maternal and child health (as part of the Ca:sh project) in India [2]. These studies lack formal evaluations of entry accuracy. Cell phones have also found broad application for mobile data collection in the developing world. Cell-Life employs electronic forms on mobile phones to improve TB and HIV treatment in South Africa [13], [33]. Electronic forms are also used by Jiva TeleDoc for improving rural healthcare in India [1, p.42], and by Pesinet for monitoring infant health in Mali [22]. Mobile phones with forms are also being used to monitor malaria in Kenya [25]; while PDAs were also piloted, the authors note that phones are more intuitive due to worker familiarity. Voxiva’s Cell-PREVEN uses interactive voice response and voice recording to monitor adverse events amongst sex workers in Peru [7]. We are unaware of any quantitative evaluation of entry accuracy in these projects. To avoid the complexities of navigating electronic forms, the CAM framework offers a hybrid system in which paper forms are used for organization while phones are used for data entry [27]. Each field on the paper form is annotated with a barcode, which is recognized by a camera on the phone

prior to data entry. Users that lacked prior camera or computer experience were trained to a level of comfort within 5 to 15 minutes. A separate study measures error rates of 1% or below using the CAM system [28]. This represents an interesting and useful design point, especially in cases where paper forms are already ingrained into the workflow. We focus on solutions that are independent of any paper workflow, and which do not necessarily require a camera-phone (while Java-phones often have cameras, our SMS and voice solutions are suitable to the most inexpensive phones). While electronic forms have been widely deployed, there are fewer solutions that rely on user-constructed SMS messages for mobile data collection. One example is a system from Dimagi, Inc. which monitors water treatment plants in India [34]. We are unaware of other systems which rely on a cue card (as we do in our evaluation) for submitting a structured SMS message to a server. Others have considered broader issues in the contextual design of user interfaces for data collection in the developing world. Examples include interface design for Auxiliary Nurse Midwives in India [14] and a methodological framework for evaluating health devices [18]. Our focus is on assessing the entry accuracy for a range of standard interfaces. III. U SER I NTERFACES Three of the central modes on a cell phone that can be used to perform data collection are voice, SMS and an electronic forms application. Data collection performed by voice can be further split into systems that link the data collector with a live operator, those that connect to an automated interactive voice response system, and those that allow the user to record a message. We focus our discussion around live voice operators, SMS and electronic form based systems, and examine some of the strengths and weaknesses of these various approaches. We use SMS to refer to data collection systems that involve information entered by a structured text message: in particular we assume that the information is entered by following a small cue sheet with a flowchart that directs the collector how to enter the data. To our knowledge, using cue cards to guide data entry by text message has not been done previously. In contrast, electronic forms (particularly on personal digital assistants) have been widely used. In this paper, we use the term “electronic forms” to denote any external application that can be placed on a phone, and that automatically guides the user how to enter data, through the use of text, menus or other tools. In a voice operator interface, the user simply calls a live operator, who asks the user a series of questions to elicit the information needed. Figure 1 illustrates each interface as used in our particular experiment.





Electronic Forms Interface
General Strengths Easy patient identification Ongoing cost is low (SMS or data plan) Can store visits when connectivity is poor


General Weaknesses Requires programmable phones Requires basic literacy skills Hard to alter survey questions Hard to enter in free-form notes Application can be deleted by user Our Results: Accuracy & Efficiency We measured 4.2 errors per 100 entries The average interaction was 99 seconds

Electronic form screenshots 1. Start application 2. Select the patient 3. Select "new visit" 4. Enter the patient's temperature 5. Select severity of patient’s cough 4 SMS Cue Card 5

SMS + Cue Card Interface
General Strengths Can be used with any phone Ongoing cost is low (SMS) Many workers familiar with SMS General Weaknesses Requires basic literacy skills Changing survey requires new cue card Hard to enter in free-form notes No confirmed receipt of data delivery Worker can forget or lose cue card Quite easy to fake visits (copy old SMS) Our Results: Accuracy & Efficiency We measured 4.5 errors per 100 entries The average interaction was 97 seconds

1. Create a new SMS Message Press Center Button Select “Messages” Select “Create Message” Select “New Short Message”

4. Enter a Space Press *


2. Switch to Numeric Input Mode Press Menu Button Select “Entry Mode” Select “Numeric”

11. Enter the Patient’s Cough No Cough - Press 1 Rare Cough - Press 2 Mild Cough - Press 3 Heavy Cough - Press 4 Severe Cough - Press 5 (with blood)

3. Enter the ID of the Current Patient Aamir Khan - Press 1 Abhishek Bachchan - Press 2 Aishwarya Rai - Press 3 …

21. Check Yourself Your finished message should be formatted similarly to the following: 10 372 62 68 4 1030007

Sample Voice Interaction

Voice Interface
General Strengths Can be used with any phone No literacy required of workers Easy to change survey questions Easy to add in free-form notes Hard to fake a visit: operator can ask new questions General Weaknesses Ongoing cost of operator salary Voice plans often higher cost than SMS Awkward 3-way social interaction Our Results: Accuracy & efficiency We measured 0.45 errors per 100 entries The average interaction was 140 seconds Operator Patient



Operator: Hello. What is your name? Worker (to operator): My name is Lipika. I am calling to record a patient visit. Operator: What patient are you visiting? Worker (to patient): What is your name? Patient: Pavathi (reading from note note sheet) Worker (to operator): Pavathi. Operator: That’s Pavathi, right? Worker (to Operator): Yes (operator records name) Operator: What is her temperature? Worker (to patient): What is your temperature? Patient: 97.1 (reading from note sheet ) Worker (to operator): 97.1 degrees. Operator: 97.1 deg. (operator records temperature) …

Fig. 1.

The three user interfaces evaluated in this paper: a) electronic forms, b) SMS + cue card, and c) voice.


In general, there are a variety of factors that affect the choice of a data collection interface. These may be loosely categorized into operation, effectiveness and cost. Figure 1 supplements the below discussion by summarizing some of the strengths and weaknesses of each interface. A. Operation We use “operation” to refer to factors involved with the general infrastructure of the data collection system. Initially there is the investment of time to set up the system, and then train the workers who will be performing data collection on the system. For voice or SMS interfaces, the set up time for workers is minimal: each worker must simply be provided with a phone, if he or she does not currently own one. However, electronic forms require that the application be downloaded onto the phone, which requires either an Internetenabled phone in an area of good connectivity, or specialized development tools and an external computer. Training time for each application is an open issue, and is one of the factors we investigate further in this study. Worker education and worker cell phone familiarity are likely to affect how easy it is to set up each user with an interface, and train them how to use it. We expect that a voice interface requires the least amount of education and background to get users equipped to start performing data collection. In particular, a voice interface does not require that its users be literate. System coverage and reliability are also critical factors to ensure good data collection. Voice calls have priority over SMS, and there is the possibility of lost SMS messages. The delivery mechanism with electronic forms can vary: both GPRS and SMS can be used. GPRS has the advantage that there is an acknowledgment of whether the data was sent; however not all locations have coverage. From the user side, voice appears to be the most reliable and has the most far reaching coverage; however, this also requires that there exists a sufficient number of operators so that users can always reach a person when they call. If this is not always possible then there may be a reliability penalty as users may have to call back later (or wait for the operator to return their call). In addition to reliability, a good system should enable some degree of flexibility. Despite good initial prototyping, it may sometimes be important to be able to modify the data collection interface, fix an error, improve usability, or add or remove information to be collected. If users have an Internet enabled phone and are always working in areas of high connectivity, then updating an electronic form system is quite feasible. However, if this is not the case, then users must reprogram their phone using the same specialized tools needed for initial set up. SMS is similarly challenging to update since a new cue card must be distributed to direct the user to enter the data. In contrast, voice is trivial to update, as the operator can simply ask a new set of questions. B. Effectiveness In any data collection effort, one of the key considerations is the effectiveness of the program at obtaining high quality

data. High quality data can perhaps be characterized by two simple criteria: whether or not the data is intentionally faked by the user, and the accuracy of data that is not intentionally faked (which is the focus of this paper). Intentionally faked data can lead to incorrect conclusions and potentially lead to significant misallocation of resources when interventions are based on false data. There may be an incentive to fake data when users are busy and collecting real data is time consuming, due to the data recording itself or transportation time to reach the source of the data (such as visiting remote patients). Unfortunately in SMS systems it is quite easy to fake data, particularly for cell phone savvy users that can copy and paste prior SMS messages. Faking electronic forms is slightly harder as it requires the user to sequentially fabricate data across an entire form. It requires the most effort for users to fake data while speaking on the phone, as the operator can always ask a new question to try to ascertain if the user is fabricating the data. Voice also has the benefit that it is easy for users to convey additional information (not included in the original survey), whereas it is more challenging to spell out text using the keypad, particularly in other languages which may or may not be supported on a given phone. Voice is also likely to have fewer operational risks: users may accidentally delete the form application, or forget their SMS cue card, but since an operator can always call a worker directly, the voice system is fairly robust. Voice also makes it easy for users to correct previous visits, by simply calling back the operator. This is also easy to do by modifying and resubmitting a saved electronic form. However, it is also important to consider the speed of data entry, how much the user likes the interface, and the accuracy of data entry. To our knowledge there are no prior studies comparing the accuracy and speed of data entry using SMS, electronic forms and voice. Since we regard these as some of the most critical factors in choosing an interface, this is a large motivation for our current study. C. Cost One of the other important considerations is cost: the most beautiful, user-friendly, accurate interface may still not be practical if the cost overhead is too high for the particular problem. Costs consist of fixed one time costs as well as ongoing marginal costs. For all three interfaces users must have a cell phone. An electronic form requires a programmable phone (such as a Java-enabled phone or Windows phone) but both SMS and voice applications can be used with any phone. The ongoing cost for an SMS phone depends on the rate per message which is typically quite low. An electronic form can send data using SMS or through a data plan; typically SMS is cheaper depending on the amount of data that is being collected. Voice minutes are frequently more expensive than SMS. But most importantly, voice has the ongoing cost of the salary of the operator, which is an additional overhead not shared by electronic forms or SMS.


IV. U SER I NTERFACE I NSTANTIATION The prior section discussed some of the general factors important to consider when designing and selecting a data collection interface. We now discuss the context for our data collection effort and the interfaces we evaluated. A. Domain context Soon the authors, along with other collaborators, intend to conduct a trial that examines whether increased information and monitoring can improve heath outcomes and adherence during tuberculosis treatment in Bihar, India. Treatment will be conducted by having tuberculosis patients regularly visit health workers and receive drugs as part of a directly observed therapy (DOT) strategy. During these visits, health workers will collect data about their patients and report this information by mobile phone back to a central office. This information will be aggregated and analyzed to inform doctors and the trial manager about which patients may need to be visited, for example, if a patient is not improving or is experiencing adverse side effects. To support this effort we need a user interface that enables fast and accurate data collection. The data collected during a patient visit will include both identification and health status information. The worker will enter in information to identify both the worker name (done only once at the start of treatment, in the case of forms and SMS) as well as the patient name. In addition, the health worker will record the patient’s current temperature, weight and pulse, as well as the presence or absence of seven symptoms: night sweats, chest pain, loss of appetite, nausea, coughing with blood, yellow eyes and fatigue. These symptoms were chosen based on advice gathered from tuberculosis health experts. The worker will also record whether the patient’s current cough is absent, rare, mild, heavy or severe with blood. The trial intervention is centered around the hypothesis that better, more frequent data collected about patients can improve tuberculosis health outcomes and therefore high quality data collection is critical. However, even if an interface encourages high quality data, it is still essential that such a data collection method also be easy to use and affordable in order for such an intervention to have widespread applicability. Originally we were planing to use electronic forms for data collection. However, since there appeared to be a dearth of literature in evaluating mobile data collection accuracy, we decided to evaluate the accuracy, speed and usability of three mobile phone interfaces. The results of this evaluation influenced our choice of an interface for use in the treatment program. B. Electronic forms implementation We created a Java application which provides a sequence of electronic forms that guide the worker to request information from the patient. The worker identification number is encoded once into the phone and is included with each recorded visit. The worker has to either enter numeric data or make a selection from a multiple-choice menu to encode symptoms.

The electronic forms underwent several design iterations, including gathering feedback from a 3-day session with 22 health workers in Bihar, India, prior to this study. Based on feedback from the workers in Bihar, we choose to employ hybrid English/Hindi menus for some of the forms, since some medical terms are easier to understand in English, but others are easier to understand in Hindi. We also changed from using multi-select lists (with a checkbox per symptom) to using individual yes/no questions. Figure 1a shows a series of screenshots of the form interface used for the present study. The Java application can be set up to either relay this information via SMS or GPRS. This distinction is important for cost considerations but does not affect the interface testing considered here. C. SMS implementation For the SMS interface we designed a cue card that instructs the worker how to record information about the patient into a text message; Figure 1b displays a subset of the cue card used. All information is coded numerically; this is done to reduce the amount of cell phone familiarity necessary, as well as to increase the speed of data entry. Participants enter in data as prompted by the cue card and then send the text message at the end of the interaction. The final part of the cue card as displayed in Figure 1b shows a sample text message. D. Voice implementation For the voice interface the worker calls a live operator. The operator asks the worker a series of questions about the patient’s health, which prompts the worker to ask the patient that question. This means that workers interact simultaneously with an operator and a patient; we are unaware of previous programs that have taken a similar approach. Figure 1c displays a sample interaction. The live operator confirms answers with the worker; this adds to the length of each call but is done to increase accuracy. This can be particularly important when the phone connection is poor or there is background noise. V. S TUDY M ETHODOLOGY The user study took place in the Surat and Bharuch districts of the Indian state of Gujarat during July and August of 2008. A. Participants As detailed in Table III, the study participants consisted of six community health workers and seven hospital paramedical staff. The community health workers were associated with the Dahej public health center; five of the paramedical staff were at the Reliance Tuberculosis hospital; and the remaining two paramedical staff were at the dispensary of the Sardar Vallabhbhai National Institute of Technology. The study participants were recruited through contacts of the first author. Initially, we had hoped to perform the study entirely with community health workers, as they are often the primary agents of remote data collection (including in our upcoming tuberculosis treatment program). However, this turned out to be infeasible because some community health workers were


unable to travel to the Dahej public health center for training and testing, and it was not feasible for us to travel to each worker’s home. This prompted us to recruit participants from two other centers. There were also some logistical challenges in performing the studies due to adverse weather conditions and the bomb blasts occurring in July 2008 in the Surat area. The education level of the health workers ranged from 10 to 12 years, while the education of the hospital staff ranged from 10 years to a B.A. degree. The average age of the study participants was 26.4 years (range 19-35). Seven participants owned a cell phone, four participants had used but did not own a cell phone, and two participants had never used a cell phone previously. Eleven of the participants were native Gujarati speakers and all spoke Hindi. B. Training Participants were trained by at least two trainers in small groups of at least two. Initially, examples were presented on a whiteboard and participants were instructed to practice entering in the data on either electronic forms or as an SMS using the cue card. After this stage, a paper with a set of example patients was handed out, and participants were instructed to practice entering in this data. In the final stage, participants were instructed to practice role playing patient– worker interactions with each other. Participants received variable amounts of training, ranging from 45 minutes to 8 hours, depending on their experience and availability. The longer training sessions were not necessarily more effective, as they were performed in larger groups. While it would have been desirable to achieve more uniform training, this was difficult given the logistics of transportation and worker schedules. Prior to the completion of training, all participants had completed at least two perfect interactions on both electronic forms and SMS, and at least one perfect interaction on the live operator mode. Throughout the user study, we employed Motorola L6i cell phones for training and testing. This is the cheapest Java-enabled phone from Motorola (the source of our current development tools) that is available in India; see Appendix A-1 for a cost analysis. All interfaces and related tools (cue cards, etc.) were presented in Hindi, and the mobile phones used had dual Hindi menus. C. Testing Participants were tested in pairs, alternating who was being tested on data entry, and who was playing the fake patient for that data point. The order of the interfaces was randomized: for a given participant pairing, the order of voice, SMS, and electronic forms was alternated. For the voice interface, the first author acted as the operator and was located outside of the room testing was being conducted in; however, there was always an additional person associated with the experiment inside the room at all times with the participants. During testing, each participant performed two complete patient–worker interactions (in the role of the worker) for each of the forms and SMS interfaces. For the voice interface, the

six community health workers completed only one interaction, while others completed two interactions (we did not anticipate that voice would become a focal point of this study until halfway through our experiments). The lag time between training and testing was exactly one day for seven of the participants, and ranged between half a day and two days for the remaining participants. All participants received a brief refresher and supervised entry session immediately prior to testing. VI. R ESULTS The results of the user study are detailed in Table III. We present both the accuracy of data entry, as well as the time needed to interview patients and report the data. On average, electronic forms and SMS offered comparable error rates of 4.2% and 4.5% per entry, respectively. The voice interface proved to be approximately 10x more accurate, with an error rate of 0.45% per entry. While only one out of thirteen participants performed perfectly on both the forms and SMS interfaces, twelve out of thirteen participants performed perfectly on voice. A Student’s two-tailed, unpaired t-test revealed that voice had a significantly lower error rate than electronic forms (p < 0.01) and SMS (p < 0.01); no significant difference was found between the error rates of electronic forms and SMS (p = 0.84). It is important to note that our results indicate a bimodal distribution of error rates: participants 7-13 performed notably better than participants 1-6. While there are many compounding differences between these participants, including the manner in which we conducted training, we refer to them by their occupation in order to simplify the discussion; participants 1-6 are health workers while participants 7-13 are hospital staff. As summarized in Table III, health workers exhibited an error rate of 7.6% for forms and 6.1% for SMS, while hospital staff exhibited an error rate of 1.3% for forms and 3.2% for SMS. In addition, the only voice error occurred with health workers. Unfortunately, our data are insufficient to explain the differences observed between these two groups of participants. On average, the hospital staff were older, more educated, and more likely to own a cell phone than the health workers. It is plausible to suspect that these factors contributed to the higher accuracy achieved by hospital staff. However, due to logistical reasons, our training procedure also differed between the two groups: health workers were trained in a large group for 6-8 hours, while hospital staff were trained in small groups for 1-2 hours. Our trainers were also somewhat more experienced when working with hospital staff, as health workers were trained first. We re-iterate, however, that training continued until all participants were able to complete two perfect trials on forms and SMS, and one perfect trial on voice. To better understand the error rates observed using each interface, we tabulate the exact sources of error in Appendix A-2. We classify errors by their entry type (numeric, multiplechoice, yes/no). We also inspect whether each error could be detected, by a trained eye, using the submitted data only; in the future, such errors could potentially be flagged or


ID 1 2 3 4 5 6 7 8 9 10 11 12 13

Occupation Health worker Health worker Health worker Health worker Health worker Health worker Hospital staff Hospital staff Hospital staff Hospital staff Hospital staff Hospital staff Hospital staff

Accuracy of Entries Time per Interaction Owns Total Used (Wrong / Total) (Average) Cell Training Cell Education Level Age Phone? Phone? (Hours) Forms SMS Voice Forms SMS Voice pre-secondary (class 10) 25 X 8 1 / 22 3 / 22 0 / 11 2:00 1:45 3:07 pre-secondary (class 10) 25 X 6 2 / 22 1 / 22 0 / 11 1:55 1:12 2:29 pre-secondary (class 10) 30 X 6 1 / 22 1 / 22 0 / 11 2:15 2:05 2:50 secondary (class 12) 19 8 2 / 22 1 / 22 0 / 11 1:33 1:27 2:34 secondary (class 12) 19 X 6 2 / 22 0 / 22 1 / 11 1:45 1:27 2:12 secondary (class 12) 20 X X 6 2 / 22 2 / 22 0 / 11 1:35 2:10 2:00 pre-secondary (class 10) 30 2.5 0 / 22 2 / 22 0 / 22 2:25 1:40 2:05 secondary (class 12) 32 X X 2 0 / 22 1 / 22 0 / 22 1:42 1:17 2:35 secondary (class 12) 28 X X 0.75 0 / 22 1 / 22 0 / 22 1:30 1:17 1:55 post-secondary (B.A.) 35 X X 1.5 1 / 22 0 / 22 0 / 22 1:25 3:15 2:00 post-secondary (D. Pharm.) 26 X X 1 0 / 22 0 / 22 0 / 22 1:05 0:55 2:10 post-secondary (D. Pharm.) 24 X X 1 0 / 22 1 / 22 0 / 22 1:07 1:25 1:52 post-secondary (M.S.W.) 30 X X 0.75 1 / 22 0 / 22 0 / 22 1:10 1:15 3:15 Average (health workers only) g ( p y) Average (hospital staff only) Average (across all interactions) Std. Dev. (across all interactions) 7.6% 1.3% 4.2% 5.9% 6.1% 3.2% 4.5% 6.4% 1.5% 0% 0.45% 2.0% 1:50 1:29 1:39 0:28 1:41 1:35 1:37 0:45 2:32 2:16 2:20 0:28


automatically fixed using self-correcting forms. Finally, we tabulate whether each error is potentially dangerous (e.g., a severe cough reported as a mild cough would prevent a physician from delivering needed care). Electronic forms witnessed errors in each entry type; only three of the twelve errors are evident from the values submitted, while five errors may be dangerous. Surprisingly, eight of the errors were due to numeric entry problems on the electronic forms. Two errors were due to a mis-placed decimal point in the temperature entry; while our interface automatically places the decimal point if needed, the user failed to enter the right number of digits in the temperature. The SMS interface also witnessed errors in each entry type; out of thirteen errors, eight are detectable and seven may be serious. Three of the errors could perhaps be averted with a revision of the SMS cue card: to indicate the absence of a patient cough, many participants entered the code “0” rather than the desired (though perhaps less intuitive) value of “1”. Unlike the forms interface, workers sometimes entered the wrong patient identity when using SMS. The voice interface witnessed only a single error for the entire duration of the trial. We consulted a videotaped record of the interaction in question (we taped one interaction for each participant), and found that the error was incurred by the operator in translating the participant’s report into a spreadsheet. While such transcription errors could indeed occur in practice, it is encouraging that the participants were not responsible for any errors on the voice interface. While the voice interface offered the lowest error rates, it also led to the longest entry times. Electronic forms and SMS averaged 1:39 and 1:37 per interaction, respectively, while the voice interface required 2:20 on average (1.43x higher than

forms and SMS). One factor that contributed to the slower entry rates using voice was the cellular coverage in our study area; the connection between participants and the operator was highly unreliable. The audio quality was frequently degraded beyond recognition, and calls were occasionally dropped and re-started. While many resource-poor environments have excellent cellular coverage (including the area of Bihar that we are planning to target with our treatment program), the weak coverage in our study area nonetheless reflects a realistic hazard of voice in some environments. In addition to quantitative results, we also solicited qualitative feedback from each participant, asking them to rank the interfaces by their order of personal preference. The forms and SMS interfaces were most popular amongst the participants, with each receiving six votes as the most popular interface. Only one participant preferred the voice interface to the others. This feedback is indicative of the poor phone connections experienced during the trial; many found voice to be frustrating due to the bad call quality. We were surprised that any participants preferred the SMS interface, given the relatively cryptic message that is produced in the end; however, participants noted that fewer keys are required under SMS than under electronic forms (which requires scrolling and selection). We also note that 8 of the 13 participants preferred the interface on which they demonstrated the fastest entry time. VII. D ISCUSSION In addition to the factors examined in our experiment, cost is a critical variable for selecting a data collection interface. For the purposes of our own decision making with regards to selecting an interface for our tuberculosis treatment program, we performed a simple cost analysis. Details are provided in


Appendix A-1, but in summary, the expected cost for data collection for each patient during his/her treatment is US $7.89 using electronic forms, US $4.59 using voice, and US $2.99 using SMS2 . These results show the cost of voice is competitive with the cost of the other two interfaces. Though SMS is slightly cheaper, in order for tracking patient symptom status to be helpful, it is essential that the reported data be close to error-free. This data will be used to guide doctor intervention, and faulty data may lead to unnecessary visits or worse, missed visits when a patient is sick. The voice interface had close to perfect accuracy and was significantly more accurate than SMS or electronic forms. Voice also allows for additional, unscripted information to be easily collected, and provides a social dimension to the health worker’s job. We anticipate that this social dimension could potentially lead to higher performance and a lower turnover rate amongst workers, since talking to an operator is likely to increase the worker’s feeling of being supported and integrated in a larger project. Voice also allows for verification to be performed easily: operators can simply request the worker to verify the data entry just given, which can be particularly useful for unusual entries. In addition, a voice interface can be replicated very easily in other contexts– no special software or cue cards need to be developed, and any cell phones can be used. While voice requires longer entry times for workers, this represents a very small fraction of their overall working day. For all these reasons, we have now decided to use a voice interface for our upcoming tuberculosis treatment program. Despite the many advantages of voice, there are still several challenges that must be addressed in practice. In our upcoming treatment program, workers will be actively examining and collecting data from patients and must report this information back to an operator. Calling the operator and keeping him on the line as the worker examines the patient may lead to a slightly awkward social interaction. Another more general challenge for voice interfaces is how to handle scenarios in which a user calls and the operator line is busy. One potential solution for these two challenges is to have the worker write down the data on paper and then call the operator. This introduces an additional opportunity for transcription errors but has the side benefit of creating a paper trail that may be used for later verification. To handle missing calls the operator could be responsible for calling back workers, or workers could leave a message that would be transcribed by the operator. An alternative solution to these challenges would be to use an interactive voice recognition (IVR) system. IVR could also be useful when there is very frequent data collection or when each survey questionnaire is very long. Hybrid live-operatorIVR systems are also possible, such as directing the worker initially to an IVR system, but automatically transferring the
2 We use Motorola phones for the electronic forms due to our current set of development tools. Moving to the cheapest available Java-enabled phone would decrease the forms cost to $5.39. However, in practice the cost of voice phones can also be reduced by leveraging existing phones in the community. The cost of voice remains competitive with forms in most practical scenarios.

worker to a live operator if the patient symptoms entered are worrisome. We look forward to exploring solutions for handling these different tradeoffs, and considering IVR solutions, as part of our future work. VIII. C ONCLUSION Given the widespread excitement in using mobile phones for collecting and analyzing data in the developing world, it is important to establish that the data entered on these devices meets the strict accuracy requirements of health, finance, and other applications. In this study, we provide a quantitative evaluation of data entry accuracy on mobile phones using electronic forms, SMS, and voice interfaces in a resource-poor setting. Our results indicate that, within the context of our study, the error rates for electronic forms (4.2% of entries wrong) and SMS (4.5% of entries wrong) may be too high to deploy these solutions in a critical application. In contrast, the accuracy of the voice interface was an order of magnitude better (0.45% of entries wrong), with only a single error observed across all trials. This result has influenced us to overhaul our plans for an upcoming tuberculosis program in Bihar, India, to switch to a voice-only interface. Employing a voice interface requires the employment of an operator, and may not be cost-effective in all countries. However, in India, the cost of this operator is more than compensated by the lower cost of voice-only handsets, voice-only cellular plans, decreased training time, and decreased literacy requirements on health workers. While this study provides an initial data point for the accuracy of data collection on mobile phones, further research is needed to distinguish the factors that are responsible for the errors observed. In the case of electronic forms, we observed error rates that are 3-8x higher than previously measured on PDAs. Our data are insufficient to diagnose whether this difference is due to the devices themselves (screen resolution, touch screen vs. keypad, etc.) or due to other aspects of the evaluation (worker education, training duration, etc.). A future study could address this question directly by evaluating PDAs and mobile phones in the same focus group. However, it is not our goal in this paper to prescribe the optimum device for mobile data collection. Rather, we aim only to highlight that there exists at least one context in which electronic forms and SMS may be too error-prone for large-scale deployment in an accuracy-critical application. In this same context, there is evidence that a low-tech alternative (voice) provides an accurate and cost-effective solution. IX. ACKNOWLEDGMENTS We are very grateful to the many people who graciously facilitated and participated in our user study. We thank the hospital staff at the Sardar Vallabhbhai National Institute of Technology in Surat, and also at the Reliance TB Hospital in Hazira. We thank Joshnaben Godia of the Taluka Development office in Vagra, Gujarat, and especially Suprava Patnaik for all her help with our study. This work was supported in part by the MIT Public Service Center.


[1] Achieving the Health Millennium Development Goals in Asia and the Pacific: Policies and Actions Within Health Systems and Beyond. United Nations Publications, Dec. 2007. [2] V. Anantraman, T. Mikkelsen, R. Khilnani, V. S. Kumar, N. R. Machiraju, A. Pentland, and L. Ohno-Machado. Handheld computers for rural healthcare: Experiences from research concept to global operations. In Workshop on Development by Design, 2002. [3] A. Bernabe-Ortiz, W. H. Curioso, M. A. Gonzales, W. Evangelista, J. M. Castagnetto, C. P. Carcamo, J. P. Hughes, P. J. Garcia, G. P. Garnett, and K. K. Holmes. Handheld computers for self-administered sensitive data collection: A comparative study in Peru. BMC Medical Informatics and Decision Making, 8:11, 2008. [4] J. Blaya and H. S. F. Fraser. Development, implementation and preliminary study of a PDA-based bacteriology collection system. Proceedings of the AMIA Annual Symposium, 2006:4145, 2006. [5] J. A. Blaya, W. Gomez, P. Rodriguez, and H. Fraser. Cost and implementation analysis of a personal digital assistant system for laboratory data collection. The International Journal of Tuberculosis and Lung Disease, 12(8):921–7, Aug. 2008. [6] Cost effectiveness study report for the PDA data capture and transmission. SATELLIFE, Sept. 2004. http://www.healthnet.org/coststudy.php. [7] W. H. Curioso, B. T. Karras, P. E. Campos, C. Buenda, K. K. Holmes, and A. M. Kimball. Design and implementation of Cell-PREVEN: A real-time surveillance system for adverse events using cell phones in Peru. Proceedings of the AMIA Annual Symposium, 2005:176180, 2005. [8] B. DeRenzi, N. Lesh, T. Parikh, C. Sims, W. Maokla, M. Chemba, Y. Hamisi, D. S. Hellenberg, M. Mitchell, and G. Borriello. E-IMCI: Improving pediatric health care in low-income countries. In CHI, 2008. [9] L. Diero, J. Rotich, J. Bii, B. Mamlin, R. Einterz, I. Kalamai, and W. Tierney. A computer-based medical record system and personal digital assistants to assess and follow patients with respiratory tract infections visiting a rural Kenyan health centre. BMC Medical Informatics and Decision Making, 6(1):21, 2006. [10] T. Engebretsen. Acceptance of information technology by health research projects in low-income countries: Intention to use and acceptance of using EpiHandy (IUAUE). Master’s thesis, Agder University College, Dec. 2005. [11] Evaluation of the SATELLIFE PDA project, Feb. 2003. Bridges.org. [12] D. Forster, R. H. Behrens, H. Campbell, and P. Byass. Evaluation of a computerized field data collection system for health surveys. Bulletin of the World Health Organization, 69(1):107–11, 1991. [13] R. Fynn. Remote HIV/AIDS patient monitoring tool using 3G/GPRS packet-switched mobile technology. In Appropriate Healthcare Technologies for Developing Countries, pages 129–138, 2006. [14] S. Grisedale, M. Graves, and A. Gr¨ nsteidl. Designing a graphical user u interface for healthcare workers in rural India. In CHI, 1997. [15] T. Groves. SatelLife: Getting relevant information to the developing world. British Medical Journal, 313(7072):1606–9, 1996. [16] P. C. Gupta. Survey of sociodemographic characteristics of tobacco use among 99,598 individuals in Bombay, India using handheld computers. Tobacco Control, 5(2):114–120, June 1996. [17] R. N. Jamison, R. H. Gracely, S. A. Raymond, J. G. Levine, B. Marino, T. J. Herrmann, M. Daly, D. Fram, and N. P. Katz. Comparative study of electronic vs. paper VAS ratings: A randomized, crossover trial using healthy volunteers. Pain, 99(1-2):341–7, Sept. 2002. [18] D. R. Kaufman and J. B. Starren. A methodological framework for evaluating mobile health devices. Proceedings of the AMIA Annual Symposium, 2006:978, 2006. [19] H. Ladd and N. Sewankambo. Uganda health information network: Technical report (submitted to the International Development Research Centre). SATELLIFE & Uganda Chartered HealthNet, Dec. 2004. http://www.healthnet.org/idrcreport.html. [20] S. O. Lal, F. W. Smith, J. P. Davis, H. Y. Castro, D. W. Smith, D. L. Chinkes, and R. E. Barrow. Palm computer demonstrates a fast and accurate means of burn data collection. The Journal of Burn Care & Rehabilitation, 21(6):559–61; discussion 558, 2000. [21] S. J. Lane, N. M. Heddle, E. Arnold, and I. Walker. A review of randomized controlled trials comparing the effectiveness of hand held computers with paper methods for data collection. BMC Medical Informatics and Decision Making, 6:23, 2006.

[22] Mali: Mobile service helps bring down infant mortality, 2007. Balancing Act News, Issue No. 364, http://www.balancingactafrica.com/news/back/balancing-act 364.html. [23] J. S. McBride, R. T. Anderson, and J. L. Bahnson. Using a hand-held computer to collect data in an orthopedic outpatient clinic: A randomized trial of two survey methods. Medical Care, 37(7):647–51, July 1999. [24] M. A. Missinou, C. H. O. Olola, S. Issifou, P.-B. Matsiegui, A. A. Adegnika, S. Borrmann, D. Wypij, T. E. Taylor, and P. G. Kremsner. Short report: Piloting paperless data entry for clinical research in Africa. Amer. Journ. of Trop. Medicine and Hygiene, 72(3):301–303, Mar. 2005. [25] Mobilizing against malaria. Expanding Horizons (by Nokia), pages 12– 13, Feb. 2007. [26] Nokia CEO message, 2008. http://www.nokia.com/A4942317. [27] T. S. Parikh. Using mobile phones for secure, distributed document processing in the developing world. Pervasive Computing, 4(2), 2005. [28] T. S. Parikh, P. Javid, S. K, K. Ghosh, and K. Toyama. Mobile phones and paper documents: Evaluating a new approach for capturing microfinance data in rural India. In CHI, 2006. [29] P. Quinn, J. Goka, and H. Richardson. Assessment of an electronic daily diary in patients with overactive bladder. BJU International, 91(7):647– 52, May 2003. [30] J. Selanikio. Advantages of using handheld computers for data collection. DataDyne. http://www.datadyne.org/files/BriefHandheldAdvantages.pdf. [31] J. Selanikio and R. Donna. Datadyne brief. DataDyne. http://www.datadyne.org/files/DataDyne brief.pdf. [32] K. Shirima, O. Mukasa, J. Schellenberg, F. Manzi, D. John, A. Mushi, M. Mrisho, M. Tanner, H. Mshinda, and D. Schellenberg. The use of personal digital assistants for data entry at the point of collection in a large household survey in southern Tanzania. Emerging Themes in Epidemiology, 4(1):5, 2007. [33] D. Skinner, U. Rivette, and C. Bloomberg. Evaluation of use of cellphones to aid compliance with drug therapy for HIV patients. AIDS Care, 19(5):605–607, 2007. [34] SMS data management. Dimagi, Inc., 2008. http://www.dimagi.com/ content/sms-data-management.html. [35] B. Tiplady, G. Crompton, M. Dewar, F. Boellert, S. Matusiewicz, L. Campbell, and D. Brackenridge. The use of electronic diaries in respiratory studies. Drug Information Journal, 31(3):759–764, 1997. [36] I. Walker, C. Sigouin, J. Sek, T. Almonte, J. Carruthers, A. Chan, M. Pai, and N. Heddle. Comparing hand-held computers and paper diaries for haemophilia home therapy: A randomized trial. Haemophilia, 10(6):698–704, Nov. 2004.



In our basic cost analysis we first assume that the treatment pool is 1000 patients. In our treatment program each worker is responsible for 10 patients, so there is a total of 100 workers. Each worker must be equipped with a cell phone. Our current development tools for electronic forms are tied to Motorola, and require a Java-enabled phone. The cheapest such phone in India is the Motorola L6i which is 75 US dollars. In contrast, both the SMS interface and voice interface can be used on any cell phone, one of the cheapest of which is the Motorola Motofone F3 ($26). Voice calls are slightly more expensive than text messages: a 3-minute voice call, which is longer than the average time in our experimental study, is about 3 rupees (0.065 US dollars, Airtel carrier). SMS messages using Airtel are 1.5 rupees per message (0.0327 US dollars). The average call length in our user study is 2 minutes and 20 seconds; therefore conducting 100 calls would require slightly under 4 hours. We therefore anticipate that a 100 call load would be reasonable for an operator working 8–9 hours per day, in order to include a liberal number of breaks. Our program design involves each worker visiting each patient to record symptom information every two weeks. At this rate a single operator


working five days per week could handle the 1000 calls over the two week period. Based on our experience in hiring a qualified operator in Bihar for $100 per month, we choose a conservative estimate of an operator salary of $200 per month. The length of treatment is six months. Given the assumptions above, we calculate the total cost per patient over the course of the treatment for each interface. Note that we are only focusing here on the aspects of the interfaces that lead to different costs and we are not considering the salary of the workers or additional program overhead. The cost of the phone per patient is simply the cost per phone multiplied by the number of workers divided by the number of patients. The cost of an operator per patient is the salary of the operator per month ($200), multiplied by the 6 month treatment length, divided by the number of patients, yielding a cost of $1.20 per patient. Workers will upload health information approximately 12 times per patient (once every two weeks). Therefore the cost of communication per patient is equal to the cost for each data entry (either SMS or a voice call) multiplied by 12. Table IV displays the cost breakdown per patient. Due to the high cost of phones that can support external applications, such as Java-enabled phones, voice is cheaper than electronic forms over a single 1000-patient program, even given the ongoing cost of an operator salary. SMS is the cheapest since it requires no operator and can be used with any phone. Perhaps most important is that the cost for each interfaces is less than $10, a small sum compared to the total cost of approximately $90–100 needed to treat a tuberculosis patient in India. A PPENDIX A-2. D ETAILED L OG
Error Number 1 2 3 7 4 5 6 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Interface Mode Forms Forms Forms Forms Forms Forms Forms Forms Forms Forms Forms Forms SMS SMS SMS SMS SMS SMS SMS SMS SMS SMS SMS SMS SMS Voice Entry Type Multiple-choice Multiple-choice Numeric Numeric Numeric Numeric Numeric Numeric Numeric Numeric Yes/No Yes/No Multiple-choice Multiple-choice Multiple-choice Multiple-choice Multiple-choice Multiple-choice Multiple-choice Numeric Numeric Numeric Numeric Yes/No Yes/No Numeric Entry Name Cough Cough Temperature Temperature Temperature Temperature Temperature Weight Weight Weight Fatigue Nausea Cough Cough Cough Cough Cough Patient ID Patient ID Temperature Weight Weight Weight Yellow eyes Fatigue Weight

Interface Forms Voice SMS

Fixed Cost $7.50 $2.60 $2.60

Marginal (Ongoing) Cost $0.39 $1.99 $0.39

Total cost $7.89 $4.59 $2.99


This cost analysis assumes that we continue to use the Motorola L6i Java-enabled phone for the electronic forms interface. There are some cheaper Java-enabled phones that we may be able to use in the future, such as the $50 Nokia 2626, but this would require us to obtain new development tools. This would change the cost per patient for electronic forms to be $5.39. This still means that voice is less expensive than forms in terms of cost per patient. Also, the cost of voice could be further reduced by leveraging existing phones belonging to the health workers. While the above analysis is conducted for a specific program in India, informal data suggests that in some other countries voice may also be worth considering. For example, the average salary of call center operators in Peru is approximately 150 US dollars per month. The biggest cost considerations when comparing interfaces in new locations are likely to be the operator salary, the cost of voice calls compared to SMS, and the expected frequency and duration of conversations between workers and the operator.

Actual Entry "none" "mild" 103.0 108.0 98 98.687 100.0 empty 67 93 No Yes "0" (disallowed) "0" (disallowed) "0" (disallowed) "0" (disallowed) missing "5" (Akshay Kumar) "1" (Aamir Khan) 103 45 826 59 "2" "000007" 59 Error Detectable? Error Dangerous? X X X

Correct Entry "mild" "heavy" 100.3 100.8 98.5 98.7 100.2 62 68 68 Yes No "1" (none) "1" (none) "1" (none) "3" (mild) "5" (severe) "6" (Akshaye Khanna) "7" (Anil Kapoor) 1003 54 62 69 "6" "0000007" 69






FOLKSOMAPS - Towards Community Driven Intelligent Maps for Developing Regions
Arun Kumar, Dipanjan Chakraborty, Himanshu Chauhan, Sheetal K. Agarwal, Nitendra Rajput
IBM India Research Lab., 4, Block C, Institutional Area Phase II, Vasant Kunj, New Delhi - 110070
Abstract— Many services taken for granted in the developed world are often missing from the developing countries. One typical example is that of map systems that form the basis of several location driven services. Its heavy reliance on content, provides a huge barrier towards building such systems. Further, in developing countries like India, the infrastructure typically has a history of unplanned development, leading to unorganized addresses and absence of standard naming conventions for roads. Detailed map systems such as online maps have only recently started becoming available but for major cities. Remote towns and villages remain out of reach till date. In this paper, we investigate a community-driven approach for creating maps in developing regions - following Web 2.0 principles, but not entirely relying on the existing Web. Our system, dubbed F OLKSOMAPS is an intelligent, community constructed map system, particularly architected with developing regions in mind. We present the design of F OLKSOMAPS, followed by an implementation of our proof-of-concept system. We present user studies aimed at understanding the uptake, usability and utility of F OLKSOMAPS. The results indicate a strong need for such a community-generated mapping ecosystem.

were offered in the manner done today. Furthermore, lack of stable infrastructure including electricity, internet connectivity and the lower purchase power of people, also complicate the situation further. The urban metropolitan cities, however, are beginning to experience such services as the factors affecting semi-urban/rural areas are not applicable there. Examples in India include MapMyIndia.com and MapsOfIndia.com. These websites provide detailed mapping information for metropolitan cities but contain only high level content for other areas. Even in metropolitan cities (about 100 million people stay in Indian metropolitan cities [2]), the (online) maps often do not contain enough detail/content to get driving directions from door-todoor. Secondly, lack of structured addressing conventions and poor road signs makes it difficult to follow the maps4 . So, even people comfortable with maps, often need to ask people on the streets to find their way. To overcome the prohibitive cost of developing and maintaining such map services for semi-urban areas, as well as to address the limitations of using maps in urban cities, we propose utilizing the collective efforts of the community who would be motivated to populate, maintain and access content for their benefit. In this paper, we present F OLKSOMAPS a community driven map system that leverages Semantic Web 5 technologies to create and manage a community generated knowledge base and makes use of web and voice applications [3] to provide access to its services. It is non-trivial to build such a system since several issues crop up. For instance, due to unplanned, historic development over centuries, cities, towns and villages in developing countries typically do not have well structured naming of streets, roads and houses. For example, postmen in villages often need to know the inmates by name in order to reach their houses. This leads to imprecise directions and key landmarks become very important in specifying locations and directions. We make use of such insights among other obtained from a user study for the design of our system.

I. I NTRODUCTION As defined in Wikipedia: “A map is a visual representation of an area’s symbolic depiction highlighting relationships between elements of that space such as objects, regions, and themes. Maps may represent any space, real or imagined, without regard to context or scale” The map systems in developed countries have advanced to a state where users can view street level information in 3D and annotate the maps with their own personalized content1 . Such systems include Google Earth2 and MSN’s Live Maps3 etc. Due to the prohibitive development cost involved, offering such systems becomes viable if profitable services can be offered on top of the core content base. The existing map systems generate revenue from services such as driving directions, finding local businesses and advertising. In contrast, sparsely populated semi-urban and vast rural areas of developing countries such as India do not have detailed map systems built for most locations. Further, the semi-literate, low income, Non-ITsavvy population [1] residing in these areas cannot use such services even if they
1 http://www.wikimapia.org 2 http://earth.google.com 3 http://maps.live.com

4 A tiny segment using GPS navigation might get along in the cities, however they become handicapped outside urban areas due to lack of maps. 5 http://en.wikipedia.org/wiki/Semantic Web





B. Results of Survey with Non-ITsavvy Subjects As expected none of the subjects in Non-ITsavvy category used maps to find information about locations. In fact, many were not even aware of the concept of maps. A majority of them mentioned that they currently rely on asking people on the street to know the location of a shop in an area. They ask people nearby or ask their friends or colleagues about the location. The others mostly find a shop on their own. 3 subjects mentioned that the information that they receive from people on the street are often not correct and not accurate, and this leads to frustration. About 87.5% of these subjects rely on other people to provide them travel directions. However, only a small section of these felt the need for precise directions (needed only in congested residential areas). About 75% people preferred the choice of a phone based interface and were willing to upload content. About 56% also mentioned that they would be willing to pay for such a service. Key Insights : Use of maps for Non-ITsavvy segment is nil and they rely primarily on others for travel directions. Most prefer a voice based interface and many are willing to pay for the call. Also, precise directions are not necessary for this user segment. C. Results of Survey with ITsavvy Subjects As shown in Table I, a majority of ITsavvy subjects make use of maps for finding locations and directions to locations. The interesting part is that even with maps, a majority of these people rely on other people for location related information. This is primarily because the unstructured nature of city layout coupled with broken, missing, faded, hidden (behind posters, graffiti) or even inconsistent sign boards make it almost impossible to rely entirely on a map for travel directions. Many a times, the maps do not contain fine-grained information in the first place. 2 out of 24 subjects mention that maps help them to find exact destination, while 19 mention that they rely on maps, only for a rough idea of the direction or rely on public transportation and mostly ask people (if road signs are not enough) for the exact location once they reach nearby their destination. Most subjects mentioned it would be helpful to have a phone-based location and direction finding system in addition to the web based interface, and about 79% expressed interest in contributing to the service by uploading content either over phone or through a web-based portal. Key Insights : Even though this segment makes heavy use of maps, they still are forced to rely on other people due to various factors. Many people prefer rough directions in the beginning (as they probably know the city) of their journey, and want detailed directions only towards the end. People rely a lot on community information, and asking people on the streets for directions and location is a common practice. III. U NDERSTANDING T ECHNOLOGY R EQUIREMENTS F OLKSOMAPS harnesses user-generated content about locations and aims to provide map-based services that repre-

In this section, we present results of survey done with endusers who depend upon location/map-based information for their daily business and/or personal needs. We investigate their current models to identify how they manage and use location information. A. Survey Process We selected the subjects considering their technical background, so that we get a good variety. We targeted two categories of people: (1) ITsavvy: People for whom computers and Internet is part of life (2) Non-ITsavvy: People who do not use computers and Internet (reasons range from them being less literate to being economically challenged) but use low-end cellphones (primarily for communication). We believe that such a mixed set would be able to provide us the right insights to evaluate the need for a community-driven map for developing regions. With the Non-ITsavvy class, we conducted the survey in a face-to-face interview mode, where specific questions were asked, albeit in a very informal and interactive manner. On the other hand, we circulated our questionnaire to the ITsavvy class and requested them to fill it up independently. The questions were targeted to understand the subject’s current model for finding landmarks and directions in the city. There were 21 questions in total. At a high level, our aim was to understand the following: 1) How do people find out points of interest (ranging from very small mom-and-pop shops to popular landmarks) ? 2) How much do they rely on maps or people on the streets? Are all their information needs satisfied by maps? 3) How do they provide location information (information about landmarks, directions) to other people ? 4) Would they be interested in consuming and producing information for a community-driven map system ? We surveyed a total of 40 subjects, with 24 from the ITsavvy category and 16 from the Non-ITsavvy category. The average age of the ITsavvy subjects was 26.37 years, ranging from 21 to 34 years. The average age of the Non-ITsavvy subjects was 32.68 years, ranging from 21 to 62 years. Most of the Non-ITsavvy subjects either did not have a formal education or primary education (10th standard) and were working in the city (security guards, car mechanics, cab drivers etc). The ITsavvy set consisted of engineers, editors, reporters, business owners etc. We summarize the key findings in Table I.
TABLE I C URRENT M ODEL : S UMMARY OF U SER R ESPONSES Responses Use maps Rely on people to find shops Rely on people for directions Want precise directions Will upload content Non-ITsavvy 0% 68.75% 87.5% 35.7% 87.5% ITsavvy 66.6% 75% 67% 58.3% 79%


sent user’s intuitive way of finding locations and directions in developing regions. We conducted an additional survey with the aim of understanding the technological requirements for F OLKSOMAPS. In this survey, we primarily focused on understanding the ways in which users express directions and location information. In addition, to provide the subjects with an idea of the system, we created a paper prototype that shows how a user would typically access the services and populate content to it. The paper prototype was an audio recording of system prompts and user responses. We created audio prototypes for two scenarios 1) User calls F OLKSOMAPS to find directions to a particular location 2) User calls to add information about a location These prototypes were created using audio recording and editing tools and were used to illustrate the concept of F OLKSOMAPS to some users. We conducted this survey along with the working model survey with a total of 40 subjects - 16 from Non-ITsavvy and 24 from ITsavvy community. Here is a sample of the questions we asked: • How do you give directions to people on the road or friends? • How do you describe proximity of a landmark to another one? • How do you describe distance? Kilometers or using timeto-travel? A. Non-ITsavvy Community When asked about how they personally give directions to anyone who asks them, 12 out of 16 subjects said they make use of landmarks to explain the direction to the destination. They use names of big roads to describe a location, and use “near to”, “adjacent to”, “opposite to” relations with respect to visible or popular landmarks to point the destination. 5 subjects said they can provide exact directions within one kilometer of the destination. 4 said that they usually give directions up to the nearest landmark thereafter which people will need to ask again. 9 subjects felt confident about guiding a person to the exact landmark. Interestingly, 6 of 16 subjects said that they use time (only) as metric to measure the distance between any two locations. 12 out of 16 subjects mentioned that they use either time or kilometers. 3 subjects mention that they sometimes use “rough” distance measures in terms of kilometers. One subject did not have the notion of kilometer as a measure at all.
TABLE II C OMMON R ELATIONSHIPS USED TO EXPRESS RELATIVE LOCATION Relationships A isNear B A adjacent to B A opposite B Non-ITsavvy 93.75% 81.25% 75% ITsavvy 67% 45.8% 75%

to guide the person. This came as a surprise to us, as we were expecting this community to be more reliant on maps. 10 out of 24 subjects usually are able to give exact direction to a known landmark using relationships as shown in table II. 13 subjects mentioned that they guide the person to the nearest big landmark and then guide the person by phone or pick the person from the landmark. 14 out of 24 subjects felt confident that people should be able to follow the way they guide. Table II shows the most commonly used relationships to describe proximity of a landmark to another one. Typically, while giving directions, such relations are used to relate a less visible or a less known landmark with a more visible or popular one. Interestingly, 21 out of 24 subjects either use both time and kilometers as a measure of distance and only 3 subjects claimed to use only kilometers to represent distance. 17 out of 24 participants never use zip codes while 6 use them rarely. This study, coupled with insights obtained from the survey of current models, has been useful to design the ontology as well as the output of F OLKSOMAPS- differentiating it from traditional map-based systems prevalent today. IV. T HE T ECHNOLOGY Based upon insights gained from the surveys we developed F OLKSOMAPS – a community generated map system. F OLKSOMAPS is designed to be populated by end users for their own consumption. This section presents the architecture and design of F OLKSOMAPS system while highlighting design choices that differentiate it from the established notion of map systems. They key differences are listed below. • It relies primarily on user generated content rather than data populated by professionals. • It strives for spatial integrity in the logical sense and does not consider spatial integrity in the physical sense as essential. For instance, information such as “Building A is located near to circle J after taking first turn on the circle while arriving from location B.” is treated complete and correct for tracing path from B to A. In other words, the direction and distance parameters are not specified in precise terms. This is because, as evident from the surveys, the end users are not likely to specify physical data while populating geographical landmarks. • A visual representation is not essential to F OLKSOMAPS which is important considering the fact that a large segment of users in developing countries do not have access to Internet. • F OLKSOMAPS is non-static and intelligent in the sense that it infers new information from what is entered by the users. • The user input is not verified by the system and it is possible that pieces of incorrect information in the knowledgebase may be present at different points of time. F OLKSOMAPS adopts the Wiki model and allows all users to add, edit and remove content freely. From the established Wikis on the Web we expect that the community would actively remove or edit invalid content and keep

B. ITsavvy Community Only 2 out of 24 subjects tell people to use maps to guide them to their house. The rest either use landmarks on roads


the maps up-to-date. However, to limit malicious intent, the system places two minor restrictions described in the next section. A. Conceptual Design We use the notion of a landmark as the basic unit of representing nodes in F OLKSOMAPS. A location represents more coarse grained geographical area such as a village, city, country etc., in addition to also representing a landmark. The core knowledgebase of the system needs to capture few key logical characteristics of locations that users are interested in specifying and making use of. These include the following: • Direction i.e. the positioning of a location relative to another one. From the surveys we found out that users are comfortable with providing relative information such as ‘towards left of’, ‘on the right side of’ etc. instead of absolute direction in the form of north, east, west, south compass points. • Distance i.e. the measure of amount of space between two locations. This can be represented as numbers along with units in which the distance is expressed. From the surveys, we learnt that F OLKSOMAPS needs to consider both time and metric units to represent distance. • Proximity and Reachability i.e. representation of information stating that one location is in close proximity to another or is reachable from another respectively. • Layer i.e. granularity of geographic area that a location name represents. It could be a division as big as a whole country or as small as a village. The notion of direction and distance from a location, are interpreted with respect to the layer that the location represents. In other words, direction and distance could be viewed as binary operator over locations of the same level. For instance, ‘is towards left of’ would be appropriate if the location pair being considered is <Libya, Egypt> or say <South Korea, Japan> but not if the pair is <Sheraton waikiki hotel, Mexico> where Sheraton Waikiki hotel is in Honolulu, Hawaii. We model the knowledgebase for representing and storing these concepts in two parts. The first one makes use of Web Ontology Language (OWL) 6 to model the categorical characteristics of a landmark, i.e. direction, proximity, reachability and layer. Use of a Semantic Web language to represent relationships between locations brings in the advantage that the system can reason on those and infer newer relationships not explicitly specified by users of the system. The second part makes use of a graph database to represent distance between landmarks which is numerical data. The two modules are used in conjunction to generate answers to queries submitted by users to the system. B. User Interaction The user interaction aspect is critical for the success of F OLKSOMAPS. This is especially true since users would tend
6 http://www.w3.org/TR/owl-ref/

to query the system either when they are stuck on road looking for directions or before starting on a trip and would be pressed for time. Further, the user set also consists of people who might be illiterate or semi-literate or not very ITsavvy. Considering these, we discuss three different modes of interaction that the system needs to support to cater to different user segments for different tasks. There are three main tasks that a user can perform with the system. First is to find a landmark/location by specifying its name possibly including some related information such as nearby places or enclosing area. Second, users can ask for tracing a path between two locations. Third, users could add to the knowledgebase by adding information about a location/landmark that they know of. In addition, some or all users may also be given the facility to edit or remove entries from the knowledgebase. We also consider three sets of users who would interact with the system. On one extreme, the users of F OLKSOMAPS are ITsavvy people who can access it over the Web. F OLKSOMAPS provides a web interface to these users for submitting queries as well as to update its knowledgebase by adding new locations and related information. On the other extreme, we have users who are illiterate or semi-literate and cannot afford to have high end devices but can use an ordinary low end phone for voice communication. Studies done earlier [3], [4], [5] suggest that a voice-based interaction works well for this user segment and for them F OLKSOMAPS supports a voice based interface for querying the system. The third segment of our users lies between the two extremes and consists of mobile people with low end devices who are familiar with SMS. F OLKSOMAPS allows SMS based querying and location updates in a constrained form for these users. C. System Architecture Figure 1 shows the architecture of the F OLKSOMAPS system. As shown, users can upload content into the knowledgebase through an SMS interface, a web based interface or through a voice interface. Similarly, the content delivery to the consumers also happens through these multiple interfaces. The knowledgebase consists of an ontology and a graph database. An ontology is used as the primary repository of the location information. This is because the user generated content cannot be expected to be complete. It is essential to be able to infer facts not explicitly populated by users in order to have a pragmatic map system. The graph portion of the database captures additional information that either cannot be expressed appropriately in the ontology or needs to be processed differently. This includes numeric data such as distances between locations. The central block of the figure forms the core of the runtime system of F OLKSOMAPS, acting as an intermediary between the consumers of the service and the knowledgebase. It consists of a module each corresponding to the tasks listed above, namely location insertion/removal, location finding and path finding. It provides a similar interface to the users across interaction modalities for information upload and retrieval.

Location Finding Module Path Finder Module Location Insertion Module Location Removal Module




Web App

Graph DB

Not shown in the figure is another concept labeled Space. It is defined as complementary and disjoint to Location. This became necessary since OWL ontologies follow the open world assumption8 which means that a relation not explicitly asserted in the ontology being reasoned upon cannot be concluded to be false since it may be specified elsewhere.
<owl:ObjectProperty rdf:about="&commgis;partOf"> <rdf:type rdf:resource="&owl;TransitiveProperty" /> <rdfs:domain rdf:resource="&commgis;Location" /> <rdfs:range rdf:resource="&commgis;Location" /> </owl:ObjectProperty> <owl:Class rdf:about="&commgis;Landmark"> <rdfs:subClassOf rdf:resource="&commgis;Location"/> <rdfs:subClassOf> <owl:Restriction> <owl:onProperty rdf:resource="&commgis;partOf"/> <owl:allValuesFrom rdf:resource="&commgis;SubArea" /> <owl:allValuesFrom rdf:resource="&commgis;Area" /> <owl:allValuesFrom rdf:resource="&commgis;Village" /> </owl:Restriction> </rdfs:subClassOf> </owl:Class>

Voice Browser

Fig. 1.

The System Architecture

In the next section, we provide some details of the design of F OLKSOMAPS knowledgebase. V. K NOWLEDGEBASE D ESIGN The F OLKSOMAPS knowledgebase consists of two parts. A graph database and an ontology of locations. The graph database is primarily a graph data structure based representation of the locations. The locations are represented by nodes and the edges between two nodes of the graph are labeled with the distance between the corresponding locations. Given the insights gained from user surveys, precise distances (and exact directions) are not key components of a map for our target users. Therefore, Graph DB is an optional component and we do not discuss it in this paper. The other, more important, part that makes F OLKSOMAPS intelligent, is the ontology of locations that helps construct paths and retrieve information that no user may have explicitly entered. This is what we describe next. A. The Location Ontology
State Division StateCapital Pargana Tehsil City Area Town

Graph API

Ontology API

Fig. 3.

OWL definitions of landmark class and partOf property


Union Territory




PartOf isA





Fig. 2.

The Folksomaps Ontology Design

Figure 2 depicts the location ontology that we created. As shown, all concepts in the location ontology derive from concept Location. The highest level concept that the ontology currently represents is Country. The rest of the concepts are defined specific to India keeping in view the administrative structure of the country 7 .
7 http://en.wikipedia.org/wiki/Subdivisions

Relationships Each location can be related to other locations to logically represent the geographical relationship that exists between them in the world. As can be observed from the figure, apart from isA relation of all locations with the Location concept, each location concept is related to one or few location concepts through a partOf relation. This relation helps establish the layering between various geographic locations, as identified in Section IV. Landmarks lie at the lowest strata of the ontology. Since the administrative structure of a country remains relatively static and is generally well known, the top part of the ontology can be bootstrapped in advance and users can be allowed to contribute instances from the lower half consisting of landmarks and sub areas. Also, some key landmarks such as historic sites of national importance and key government offices such as the parliament could also be pre-populated into the ontology. The ontology supports several relationships in order to be able to specify the four key characteristics of a location defined in Section IV. In the interest of space we present here details of only two key relationships – nearTo and connectedTo that model the proximity and reachability characteristic respectively. Relation nearTo is a symmetric relation defined between two locations to express the fact that they are in close proximity to each other. This could intuitively mean a few hundred meters or a kilometer. The logical integrity of nearness can be applicable to locations other than landmarks. For example, two towns in the same district can be considered nearTo each other, compared to two towns in different districts. To capture these, for SubAreas a sameTown relation and for Areas a sameDistrict relation and so on are defined. However, for the purpose of this paper, we stick to the basic nearTo relation between landmarks. All the modules make use of this relation
World Assumption

of India

8 http://en.wikipedia.org/wiki/Open


to search for or add/update locations specified in user’s query. The connectedTo relation is a symmetric as well as a transitive relation. It expresses the fact that two locations are reachable from each other via one or more paths that can obtained from the ontology facts. Users may add connectedTo relation between location instances that they are familiar with. F OLKSOMAPS makes use of the ontology to infer new connectedTo relations based upon these individual assertions. The Path Finder module relies on this information to compute paths from the individual connections supplied by users. Bootstrap Process The F OLKSOMAPS system could be bootstrapped from existing databases to populate instances of location types in the upper part of the ontology. Two such sources of data in the absence of a full-fledged Geographical Information System (GIS) system come from the Telecom Industry and the Postal Department. While postal department is obvious, the telecom companies also maintain database of various circles that they operate in. Given that mobile phones have penetrated into the remote rural areas as well, the upper ontology can be populated from their data as well. While the actual GIS data benchmarking require significant efforts and cost on the field to map the spatial data, the telecom and post office data give a very good logical view of the locations. This complements our system’s design goal of providing a logical view rather than a spatial view to the users.

new entry is rejected. Also, this landmark is added only within the context of its parent, i.e. it’s SubArea. If the SubArea supplied does not exist, then also the landmark is rejected. In addition to the landmark name, the user can optionally specify other meta information such as which all landmarks are located nearby and to which all landmarks is this new one is connected to. 5) editLandmark() : This method allows you to search for a landmark or a relation instance and allows you to delete it in the fashion similar to addLandmark(). In the next section, we describe our prototype implementation that we used to conduct user studies. VI. P ROOF - OF -C ONCEPT I MPLEMENTATION We have implemented a prototype of F OLKSOMAPS and deployed it at our lab. The prototype’s knowledgebase includes the ontology module and does not have the optional Graph module. We used OWL to implement the ontology. We implemented the F OLKSOMAPS modules for finding a location, finding a path and adding a landmark. The ontology API used is JENA9 with Pellet reasoner10. We bootstrapped the system with data about New Delhi, the capital of India and initialized it with SubAreas located in a couple of Areas under South Delhi district. We implemented a Web based interface as well as a Voice based interface for this prototype. The Web based interface supports all the implemented modules and is developed using Java Server Pages (JSP)11 . On the other hand, the Voice based interface is accessible over a phone call and supports find location and find path modules. It is developed using JSPs and VoiceXML12 . Users are allowed to populate the F OLKSOMAPS system with new landmarks and associate them to the SubArea which they belong to. Additionally, users can also provide information about the landmark. This includes other landmarks located near to the one being added, and other landmarks that are connected to this by road etc. Figure 4 shows a partial snapshot of the populated Folksomaps knowledgebase.
New Delhi (Capital)

B. Knowledgebase API This subsection describes the API that we built for accessing the knowledgebase for finding path, location or for adding a landmark. 1) findLocation() : This method allows a user to search for a location specified by its name. Optionally, extra information can be supplied which includes the landmark’s relationship with another landmark or its attributes. 2) findPath() : This method takes source location name and destination location name along with a filtering criteria and returns a list of locations that represent the path traversal from source to destination. The location names can optionally be augmented with a list of nodes that represent meta information about the position of the source or destination node in the ontology. The filtering criteria specifies additional restrictions (e.g. only traverse paths connected by a nearTo relation) on the path traversal algorithm. 3) doesExist() : This method determines whether the supplied landmark name already exists in the knowledgebase. 4) addLandmark() : This method allows the user to insert a new landmark into the knowledgebase. It takes the name of the new landmark and also its immediate parent, i.e. its SubArea name. Optionally, its next parent, i.e. PostOffice is also supplied. If this landmark already exists, this

South Delhi (District)

Munirka (Area) VasantVihar (Area) Mahipalpur (Area)

VasantKunj (Area)

Malvianagar (Area)

Saket (Area)

JNU VasantLok Market

PVRPriya Badam Market

Sector B (SubArea) Anchal Plaza ISID GDGoenka School

Sector C (SubArea)

PVR Saket
User Populated Landmarks

partOf connectedTo isNear
Country/Capital/District /Area/subarea

GoldenDragon Hotel Sarvapriya Vihar



Fig. 4.

A partial snapshot of populated ontology

11 https://java.sun.com/products/jsp

10 http://pellet.owldl.com

9 http://jena.sourceforge.net/

12 http://www.w3.org/TR/voicexml20/

Bootstrapped Data


Users are also allowed to query F OLKSOMAPS for getting location information and directions. Figure 5 shows the screenshot of webpage for getting directions from a source to destination location. Figure 6 shows the flowchart for querying through the Voice based interface. Error steps are not shown in this flowchart. The results from F OLKSOMAPS reflect the ways in which people would essentially give directions.

website. In addition to the bootstrapped data, the knowledgebase also contained some pre-populated locations. We requested them to populate content into F OLKSOMAPS while restricting the locations to a set of 6 Areas in South Delhi District. This was done so that the content populated does not get thinly spread out and is relatively rich for querying 14 . After populating some landmarks known to them, users then queried the system for finding information about other landmarks and travel directions to those. After this, we asked them a few questions about their view of F OLKSOMAPS system. For the Non-ITsavvy subjects, we briefed them about the purpose of the proposed system and gave an explanation of the prototype. We then walked them through the voice based interface, by querying for some location and requesting for a path to that location from another. This was followed by a question answering session. All subjects grasped the concept fairly quickly and were able to see the benefits they could derive from such a system. B. Results of Survey for Non-ITsavvy subjects We conducted a total of 22 surveys with Non-ITsavvy participants using the voice interface. The set of interviewed people consisted of porters, security guards, elderly people, draughtsmen, waiters and service staff. The results are tabulated below.

Fig. 5.

Web UI showing the results of querying for directions

Welcome to Folksomaps

Source Location?


Find 'Location' or 'Path' ?


Location Name? Play Location Details

Get Src_Name

Get Loc_Name

Destination Location?

Sample User Interaction System: Welcome to Folksomaps. Using this service you can find information about locations in Delhi. To find information about a location say Information, to find a route between two locations say Route User : Route System: Please speak the source location User : IIT System: Please speak the destination location User : ParkBalluchiRestaurant System: You can go from IIT to SDA. From SDA you can go to GreenParkMarket From GreenParkMarket you can go to ParkBalluchiRestaurant

Get Dest_Name

Play Path Details

Fig. 6. Voice UI flow for querying F OLKSOMAPS. Red dots indicate voice recognition steps.

VII. S OLUTION S URVEY In this section, we present results of user studies we conducted to verify the benefit and acceptability of the proposed system. We further present insights that we obtained from users while conducting the survey. For the ITsavvy segment, we let the subjects try F OLKSOMAPS through the web based user interface of the system. For the Non-ITsavvy segment, we conducted the survey with the voice based interface of F OLKSOMAPS13 . A. Survey Process For ITsavvy survey participants, we started with a small introduction before giving them access to the F OLKSOMAPS
13 The voice based interface was in Hindi language and allowed a restricted set of landmarks to keep the speech recognition accuracy high for the prototype

Fig. 7.

Interviewing the Non-ITsavvy users

TABLE III S UMMARY OF N ON -IT SAVVY U SER R ESPONSES TO F OLKSOMAPS Questions Would you call to get directions? Prefer calling over asking people on the street? Ready to pay for call (else want ads)? Will upload content? Voice Interface preferred over SMS? Results need to be very accurate? Yes 100% 82% 45% 73% 91% 86% No 0% 18% 55% 27% 9% 14%

As can be seen from Table III, all the subjects surveyed were interested in using F OLKSOMAPS system. Most of them preferred the option of calling up a number for directions rather
14 We envision that the actual content in a deployed system will be much more richer than the content populated by the survey participants.


than asking someone on the street. Apart from convenience, this overwhelming response can be attributed to a couple of reasons. First, asking on the street does not always work. On a secluded road, there may not be anyone to ask. Several times people end up giving wrong directions (possibly to hide their lack of awareness or to not appear rude) leading to precious time spent traveling on a wrong route and recovering from it. Also, the point of view of street vendors who often get enquiries from passersby about directions is quite interesting. They candidly admitted that during the course of the day they get so many such queries for detailed travel instructions that it is easy to get irritated and shrug them off. Second, an important insight provided to us by our subjects was the sense of security that they would get with such a system. We were informed that even though asking for travel directions from strangers on the street is an option, it exposes the enquirer to criminal elements, who often take advantage of their lack of knowledge and use it to rob them, the activity sometimes resulting into a worse situation such as a murder. This is especially true for first time visitors to the city from remote towns and villages or female citizens traveling at odd hours of the day. Many people were willing to pay for the call even a small premium over normal charges as they saw value to having this information available to them at all times. Yet a majority of the subjects preferred the advertisement model where an advertisement played in the beginning of the call pays for the entire call. This is understandable, given this segment’s high sensitivity to cost. A few users suggested that the advertisement model was better since most of these users primarily have pre-paid SIM cards and often they do not have sufficient balance to make outgoing calls. Most people were willing to upload content, though a few refused as they were hesitant due to not owning a phone. Almost everyone preferred the voice based interface over SMS even though we demonstrated speech recognition errors during the study interviews. The primary reason for this cited by them was that many people are either not comfortable using SMS or not comfortable using a mobile phone itself. However, some users who were well versed with SMS preferred it over voice. In terms of accuracy of returned results, most people asked for full accuracy while a very few were okay with minor mistakes. The need for strong accuracy is driven by the fact that most of these people either use public transport, or use a bicycle or even walk to reach their destination. The cost of a wrong input for them is huge compared to a person driving in his own or rented vehicle. In fact, one of the main reasons for preferring a voice call over asking people for directions was to avoid wrong directions. This is an important feedback since we started with the assumption that we do not need strict controls over the content and the wiki model would work. But the tolerance for incorrect information is low and we need to factor this in. We also learnt that meta information is as important to NonITsavvy users as the landmarks themselves. For instance, in

cities, more than the road route from a source to destination, people from the underprivileged segment were more interested in knowing the bus route numbers that could take them to their destination. Road routes serve well those people who travel by their own vehicles but the underprivileged rely primarily on public transportation. Similarly, for rural areas that consist of remotely located towns and villages, what helps the underprivileged people is information regarding modes of transportation (train, bus, boat, cycle-rickshaw, taxi etc.) to take from source to destination, where to make a switch and estimated travel time. Time tables of these public transport mechanisms are another important feature for this segment that can become an essential part of F OLKSOMAPS. Key Insights : We realized that accuracy of the information is a key requirement and more the meta information available, merrier it would be for these consumers. Also, voice based interface is indeed a preferred mode for this user segment over SMS and the calls to the system should to be free of cost. C. Results of Survey of ITsavvy subjects For ITsavvy segment, we conducted the survey with a total of 15 subjects using the web based interface. We also told them that the system has a voice based interface available over a phone call and supporting similar API. The user list consisted primarily of software professionals apart from a couple of businessmen. As expected, the ITsavvy community had significant experience in using the current online maps of cities in Indian metropolitan cities and were able to carefully evaluate our approach, considering the map services that are already operating in metropolitan cities. We try to capture learnings from their feedback.
TABLE IV S UMMARY OF IT SAVVY U SER R ESPONSES TO F OLKSOMAPS Questions Would you access it for directions? Prefer F OLKSOMAPS over asking people? Ready to pay for call (else want ads)? Will upload content? Prefer Web for upload? Results need to be very accurate? Yes 93% 87% 67% 87% 92% 53% No 7% 13% 33% 13% 8% 47%

As is evident from the results (Table IV), most survey participants mentioned that they would like to use this service and that it would certainly be more convenient than asking people around in the streets. Interestingly, a bulk of the ITsavvy community did not stress on getting fine-grained direction all the time. They were fine with getting high level directions involving major landmarks. Most people were fine with paying for the service when offered on phone. Most were also willing to upload content into F OLKSOMAPS but preferred to do so over a web based interface as opposed to SMS or a voice based interface. Accuracy of responses was important for this segment as well, though not as strongly as for the Non-ITsavvy segment. As discussed earlier, this segment typically used their own vehicles and need high level directions rather than precise route.


A few participants pointed out that while voice-based access is good for interactive session, they would still prefer the content is sent to them via SMS so that they can store it for future access, pointing out that it is easy to forget the directions if you just hear it. However, this segment had other expectations from the system. Some subjects mentioned that the system should adapt to the user’s request and have the ability to produce fine-grained details depending on whether the destination is a popular landmark, or an area or a house in a colony. Few others mentioned that the directions provided by F OLKSOMAPS should take into consideration the amount of knowledge the subject already has about the area, i.e. it should be personalized based upon user profile. One subject mentioned that the current prototype appeared more suitable for driving directions but not for pedestrians. Just like the Non-ITsavvy community, the ITsavvy community also reflects the need for meta information on such a community-driven map. One subject mentioned that he would be interested in uploading traffic information on routes and would benefit from community uploading such information on the system. A few subjects mentioned that frequent changes in road plans due to constructions should be captured by such a system - thus making it more usable than just getting directions. Key Insights : While accuracy and convenience score with IT-Savvy population as well, this segment turned out to be more ambitious in terms of deriving benefits from such a system. Based upon the feedback listed above, we learnt that the user interfaces of F OLKSOMAPS needs to be rich and adaptive to the information needs of the user when considering this community. It also appears to the authors that dynamic and real-time information augmented with traditional services like finding directions and locations would certainly add value to F OLKSOMAPS. VIII. D ISCUSSION F OLKSOMAPS we believe, presents a novel approach towards developing a self-sustaining map system, harnessing community input, particularly targeted towards developing countries, where there is a need for such a system. Reasoners using ontologies consume space and compute power. [6], [7], [8] reports ways through which spatial reasoning can be made faster. This is an issue with F OLKSOMAPS as well. However, F OLKSOMAPS as a system compliments this body of work as its focus is not on improving reasoning capabilities or address scalability needs of underlying ontology reasoners. Rather F OLKSOMAPS can benefit from this body of work by adopting the solutions suggested to improve the computational and reasoning efficiency. Given their preference towards voice based interface over SMS [9], [10], designing an efficient and user-friendly voicebased user interface for the masses is important for F OLK SOMAPS . For example, while finding directions, user interface should be designed in a way that facilitates users to specify the level of detail they are looking for, varying from source to the

destination. Voice interface also takes care of the language barrier since content can be delivered in local language as demonstrated in other systems [3]. Voice based interfaces are, however, constrained with the capability of speech recognition technology which is under slow but constant improvement. Our surveys indicate that most people would like to contribute to F OLKSOMAPS knowledge base. However, in reallife, there are several factors that provide impedence for a user to be an effective information producer. Reasons range from users becoming busy, loosing interest after an initial surge, etc. In a live deployment of F OLKSOMAPS, one needs to also consider pragmatic business models (such as bartering models, advertisements or incentives) using which an in-flow of information can be sustained to keep improving the quality of system responses. Accuracy of results being of primary concern universally, further research is needed to ensure that various modules of F OLKSOMAPS would guarantee correct and precise results given that the data input by the users is correct in the first place. Also, as the knowledgebase as well as the userbase grows, established scale-up techniques would have to be applied for real life environments. IX. R ELATED W ORK Two research areas that are very relevant for F OLKSOMAPS are the areas of research in use of IT for underprivileged in developing regions and semantic tools for geographic information systems. Apart from these, work in the area of intelligent user interfaces for masses in developing regions is also relevant. There is a lot of literature on means to harness available information and user generated content [11], [12], [13] to deliver useful services to underprivileged in developing regions. [14], [15] talks about voice-driven technologies (e.g. audio wiki) to capture user content from Non-ITsavvy masses. The Neighbourhood Mapping [16]15 initiative proposes the involvement of school students to gather community input in the context of building maps. The project used PDAs coupled with GPS to build an information repository that could be used of planning purposes. F OLKSOMAPS builds further along this direction and proposes to create alternatives to well established solutions in developed countries, solely through user generated content. Specifically, it focuses on creating a framework (exploiting ontological reasoning), where by geographic information can be captured, enriched, and funneled back to the masses - customized to the needs of developing regions. To the best of our knowledge, we are not aware of any such community-driven map system for developing regions. There has been considerable work on Place ontologies, retrieval and storage of geographical information using ontologies [8], [17], [18], [6], [7]. For example, [8] talks about limitations of OWL to support spatial reasoning, integrity rules, and proposes a combination of spatial data-based store
15 http://www.csdms.in/NM/


and ontology-based reasoning to better represent geographic information and resources. [6] further critically evaluates ontology-based approaches towards geographic information retrieval while [7] presents a spatially aware search engine, for semantic interoperability of distributed and heterogeneous GIS on the Internet. In contrast to this literature, F OLKSOMAPS does not focus on improving reasoning capabilities of OWL or address spatial and logical integrity issues of Place ontologies. Rather, we focus on designing concepts of an ontology that is suitable for capturing map information from communities, keeping only logical integrity in mind, and by exploiting currently standardized semantics of OWL. F OLKSOMAPS in a way is hence complimentary to this body of work and can benefit from it. A body of research focuses on intelligent user interfaces for developing regions [19], [10], [20] and infrastructures to take IT services to the economically challenged and NonITsavvy masses in developing regions [3], [21]. F OLKSOMAPS at its core can certainly benefit from user interface designs to facilitate upload and download of map information. The system, by having multi-modal front-ends fits well with the architectural principles outlined in [3], [21]. X. C ONCLUSION In this paper, we investigated feasibility of a communitydriven approach towards creating maps for developing regions. Our system dubbed F OLKSOMAPS, has the potential of providing an effective alternative to expensive map solutions using community input, making map-based services (finding directions, finding locations and landmarks) available to people in developing regions where such services are currently missing. F OLKSOMAPS builds on the current models adopted by users in developing regions and leverages their collective knowledgebase thus overcoming the huge cost barrier in developing such a system. It is specifically designed to provide content that is intuitive for the users. We conducted a total of about 77 interviews in the process of evaluating a need for such a system and testing our prototype for verifying usability and utility of F OLKSOMAPS. Our surveys suggest that the community is very receptive towards the concept of a community-driven map as that alleviates some of the problems (reliance on people, security, inaccurate directions, etc) they face in day-to-day life. We intend to build further upon this system and overcome its current limitations to bring it even closer to users’ expectation. ACKNOWLEDGMENT The authors would like to thank Amit Nanavati for suggesting the ’Folksomaps’ name. R EFERENCES
[1] “The Use of ICTs by Small and Informal Businesses,” http://research.microsoft.com/ jdonner/ Papers/donner micros slides.pdf. [2] “World Gazetteer. India - Metropolitan Areas,” http://www.worldgazetteer.com/wg.php?x=&men=gcis&lng=en&dat=80&geo=104&srt=pnan&col=aohdq&msz=1500&va=&pt=a.

[3] A. Kumar, N. Rajput, D. Chakraborty, S. Agarwal, and A. A. Nanavati, “WWTW: A World Wide Telecom Web,” in ACM SIGCOMM Workshop on Networked Systems For Developing Regions, Aug 2007. [4] M. Plauche and M. Prabaker, “Tamil Market: A Spoken Dialog System for Rural India,” in Working Papers in Computer-Human Interfaces (CHI), 2006. [5] T. S. Parikh, “Mobile Phones may be the Right Devices for Supporting Developing World Accessibility, but is the WWW the Right Service Delivery Model?” in International Cross-Disciplinary Workshop on Web Accessibility (W4A), Scotland, May 2006. [6] A. Abdelmoty, P. Smart, C. Jones, G. Fu, and D. Finch, “A critical evaluation of ontology languages for geographic information retrieval on the Internet,” Journal of Visual Languages and Computing., pp. 331– 358, August 2005. [7] M. Kun and B. Fuling, “An Ontology-Based Approach for Geographic Information Retrieval on the Web,” in International Conference on Wireless Communications, Networking and Mobile Computing, Sept 2007, pp. 5959–5962. [8] A. Abdelmoty, P. Smart, and C. Jones, “Building Place Ontologies for the Semantic Web: Issues and Approaches,” in Proceedings of the 4th ACM workshop on Geographical Information Retrieval, Lisbon, Portugal, Nov 2007. [9] “Africa:Impact of Mobile Phones,” Vodafone Policy Paper Series, Mar 2005. [10] A. Kumar, N. Rajput, D. Chakraborty, S. Agarwal, and A. A. Nanavati, “Voiserv: Creation and delivery of converged services through voice for emerging economies,” in In Proceedings of the WoWMoM, Finland, June 2007. [11] R. Abraham, “Mobile Phones and Economic Development: Evidence from the Fishing Industry in India,” in IEEE/ACM International Conference on Information and Communication Technologies and Development (ICTD), Berkeley, USA, May 2006. [12] K. Ramamritham, A. Bahuman, C. B. S. Duttagupta, and S. Balasundaram, “Innovative ICT Tools for Information Provision in Agricultural Extension,” in IEEE/ACM International Conference on Information and Communication Technologies and Development (ICTD), Dec 2006. [13] P. K. Reddy, G. Ramaraju, and G. Reddy, “eSaguTM: A Data Warehouse Enabled Personalized Agricultural Advisory System,” in ACM International Conference on Management of Data, China, June 2007. [14] P. Kotkar, W. Thies, and S. Amarasinghe, “An Audio Wiki for Publishing User-Generated Content in the Developing World,” in HCI for Community and International Development (Workshop at CHI 2008), Florence, Italy, April 2008. [15] L. Wang, P. Roe, and B. Pham, “An Audio Wiki Supporting Mobile Collaboration,” in Proceedings of the ACM Symposium on Applied Computing (SAC), Brazil, March 2008. [16] R. Mallick, H. Kalra, and D. Banerjee, “Infusing map culture through participatory mapping,” GIS@development. http://www.gisdevelopment.net/magazine/years/2005/feb/infusing.htm, Feb 2005. [17] J. Kay, W. Niu, and D. Carmichael, “ONCOR: Ontology and Evidence based Context Reasoner,” in Proceedings of International Conference on Intelligent User Interfaces (IUI), Honolulu, Hawaii, Jan 2007. [18] G. Look and H. Shrobe, “Towards Intelligent Mapping Applications: A Study Of Elements Found In Cognitive Maps,” in Proceedings of International Conference on Intelligent User Interfaces (IUI), Honolulu, Hawaii, Jan 2007. [19] I. Medhi, A. Sagar, and K. Toyama, “Text-Free User Interfaces for Illiterate and Semi-Literate Users,” in IEEE/ACM International Conference on Information and Communication Technologies and Development (ICTD), Berkeley, USA, May 2006. [20] J. Sherwani, S. Tomko, and R. Rosenfeld, “Sublime: A Speech- and Language-based Information Management Environment,” in In Proc. ICASSP, May 2006. [21] A. Kumar, N. Rajput, S. Agarwal, D. Chakraborty, and A. A. Nanavati, “Organizing the Unorganized - Employing IT to Empower the Underprivileged,” in Proceedings of 17th ACM International Conference on World Wide Web(WWW), Beijing, China, April 2008.


HIV Health Information Access using Spoken Dialogue Systems: Touchtone vs. Speech
Aditi Sharma Grover, Madelaine Plauché, Etienne Barnard, Christiaan Kuun
Based on this perceived value of SDSs in the developing world, a number of exploratory studies have been performed in recent years. Barnard et al. [5] report on preliminary experiments performed to assess the usability of a telephonebased information service for access to government information in South Africa. A kiosk-based SDS for agricultural information was developed by Plauche et al. [6], and evaluated with semi-literate users in rural Tamil Nadu, India. Nasfors [7] also developed an agricultural information service, aimed at mobile telephone users and deployed in Kenya. The most sophisticated speech technology in this category was employed in the telephone-based information service for community health workers developed by Sherwani et al. [8]; this was piloted in Sindh, Pakistan. Agarwal et al. [9] have implemented a telephone-based kiosk system, which they call “VoiKiosk”; this is being trialled in rural villages in Andhra Pradesh, India. Each of these pioneering studies was primarily aimed at assessing the feasibility of using speech technology in various settings in the developing world. However, in the process of determining feasibility, a number of practical lessons were also learnt. For example, it was found that user acceptance of such systems is proportional to the difficulty that users would have to access the same information through other mechanisms [6] (thereby confirming the concept of the “motivated user”), and two studies [5, 8] found that it may be preferable to use more verbose, less efficient user interfaces to guide inexperienced users for whom time pressure is not a primary concern. The current contribution similarly has a twofold aim: both to explore the use of an SDS in a new environment (namely, by caregivers of HIV positive children in Botswana, Southern Africa), and to contrast different input modalities used in such a system. In particular, we compare systems using key-presses (“DTMF or Touchtone”) with those that use automatic speech recognition (ASR) for user input in this application. To this end, we start with the most basic variant of speech systems (key-press replacement), and compare such systems with DTMF input. The motivation for this choice is twofold: on the one hand, key-press replacement is likely to be more acceptable in the developing world, where general numeracy is less common; on the other, such systems are much easier to develop than natural-language systems, and are therefore more attainable in the resource-constrained environments that typically characterize the developing world. To our current knowledge DTMF and ASR input modalities have not been compared systematically in the developing world. However, in a review of developed-world applications

Abstract— This paper presents our work in the design of a SDS for the provision of health information to caregivers of HIV positive children. We specifically address the frequently debated question of input modality in speech systems; touchtone versus speech input, in a new context of low literacy users and a health information service. We discuss our experiences and fieldwork which includes needs assessment interviews, focus group sessions, and user studies in Botswana with semi and low-literate users. Our results indicate user preference for touchtone over speech input although both systems were comparable in performance based on objective metrics. Index Terms— Spoken dialogue systems, DTMF, Touchtone, Speech interfaces, Health information, HIV, Illiterate users, Semi-literate users, Low literate users, ICT, Developing regions, Information access, Africa.


There is a widespread belief that spoken dialogue systems (SDSs) will have a significant impact in the developing world [1]. This belief is based on a number of factors. Firstly, illiteracy is predominantly a problem of the developing world (according to a recent estimate, about 98% of the illiterate people on earth live in the developing world [2]), and speechbased access to information may enable illiterate or semiliterate people to participate in the information age. Also, the availability of traditional computer infrastructure is low in the developing world, but telephone networks (especially mobile/cellular networks) are spreading rapidly [3]. (For example, a recent community survey in South Africa [4] found that 73% of households owned at least one mobile phone, but only 7% of homes had internet access.) A further factor is the strong oral culture that exists in many traditional societies, which is likely to render such systems more acceptable than text-based or graphical information sources. Finally, the availability of relevant services and alternative information sources is often low in the developing world.
Manuscript received September 22 2008. This research was supported by OSI, OSISA and the NRF under the Key International Research Capacity (KISC) programme, UID no. 63676. (Corresponding author) A. Sharma, E. Barnard and C. Kuun are with the Human Language Technologies (HLT) Group in the Meraka Institute, CSIR, Pretoria, South Africa. (phone: +27128413028, email: <asharma1, ebarnard, ckuun>@csir.co.za) M Plauché was a visiting researcher at the Meraka Institute during the project. (email: mad@brainhotel.org)


comparing DTMF to ASR input by Lee and Lai [10], both user preference and performance are found to depend on the nature of the task, the personality of the user, and the capabilities of the speech-recognition system. With the exception of only one call-routing system [11], all studies found that simply replacing key-presses with speech does not improve user performance or perception. Conversely, wellengineered speech recognition systems with natural-language input are often preferred to DTMF for tasks that are not easily accomplished with DTMF. In terms of user preference, studies in laboratory settings [10], [12] report that users find speech input more interesting and enjoyable to use, but contrastingly in a recent informal poll of over a thousand users of realworld information-access systems [13], almost half of the users responded that they would prefer to use a speech input modality “as little as possible” and only 8% would do so “most of the time”. Despite these reservations, numerous applications that are completely reliant on speech input are currently in use in the developed world - examples are the health-management systems described by Migneault et al. [14] and commercial voice portal systems, such as the Tellme portal (which serves 40 million phone calls per month, according to its providers [15]). Below, we first provide background on the health-care application selected for our study and describe the system that was developed as well as the experimental protocol employed (Section II). Section III contains our experimental results, including user profiles, usability measurements and task completion rates. Finally, we discuss the scope and generalizability of our results, and conclude with thoughts on next steps to be taken along this research trajectory. II. OPEN PHONE: HIV/AIDS HEALTH INFORMATION LINE IN BOTSWANA A. Background HIV/Aids is perhaps the gravest health pandemic to face the world, and Southern Africa has been the worst hit region. Of the 33 million people infected with HIV worldwide, approximately two-thirds are inhabitants of sub-Saharan Africa [16]. Within sub-Saharan Africa, Botswana has one of highest HIV prevalence rates, with 1 in every 4 adults being HIV positive [16]. The hardest hit in Botswana are women; nearly 40% of pregnant women (ages 25-39) are living with HIV and infection levels are increasing amongst pregnant women aged 30-34 years, with nearly one in two living with HIV [17]. Aids deaths have orphaned approximately 120 000 children (ages 0-17) and another 14 000 children (aged 0-14) are living with HIV in Botswana. During our study a partnership was established with The Botswana-Baylor Children’s Clinical Centre of Excellence (hence forth referred to as Baylor). Baylor is a specialised paediatric institute serving the Botswana capital, Gaborone and its neighbouring areas since June 2003. The centre is staffed collaboratively by U.S. and Botswana health professionals. The services provided by Baylor range from

primary and specialty medical care for HIV/Aids, to catering for the psychosocial needs of HIV patients and their families [18]. Baylor provides treatment to over 2100 children infected with HIV and 260 families across Botswana. The centre is involved in a number of support activities such as community outreach programmes for patients in rural areas, servicing 20 communities outside Gaborone and a “Teen Club” to provide moral support and counselling to teenagers living with HIV [18]. A child is typically brought to Baylor to be tested for HIV/Aids by a caregiver. A caregiver is any individual who takes care of an HIV positive child; it may be the child’s parents (who themselves might be HIV positive), other family members or an unrelated community member. Children who test as positive receive free treatment from Baylor for the remainder of their infancy and adolescence. Caregivers of such children are also counselled and trained. Baylor provides free lectures for caregivers three times a week, where many aspects of HIV/Aids, antiretroviral (ARV) medication are explained and advice is given on how to live with the condition. Each caregiver on average attends two lecture sessions. The primary focus of the lecture sessions in Baylor is on adherence to ARV medication, with topics such as the principles of HIV, universal precautions, basics of ARV therapy, medication dosage, side effects and storage, and importance and strategies for adherence, being covered.

Fig. 1. Baylor in Gaborone, Botswana.

B. Open Phone Development Open Phone is a pilot HIV/Aids community-oriented SDS service that makes use of language technologies to address acute informational needs of caregivers of children with HIV. 1) Preliminary Investigations Our initial investigations started in April 2007, where we conducted interviews and discussions with 2 doctors, 4 nurses, 9 caregivers, a social worker and the technical manager. We also accompanied 3 community outreach workers on visits to 2 caregivers’ homes. The intention of this investigation was to identify specific needs of the various user groups and their day-to-day tasks.


This trip also served to acquaint us with HIV domainspecific terminology used by the interviewees to describe their situations, tasks and other notable work processes. These ‘hidden’ pieces of information allowed us to build a profile of the individuals interviewed and make informed choices later in the process of application design. This initial field trip highlighted a number of challenges and issues for consideration in providing health information to caregivers: Caregivers often struggle to recall material covered in the lecture sessions. Caregivers often have questions regarding general health information, for example, how to deal with infections, nutritional needs, hygiene requirements and commonly held misconceptions about HIV. Travelling to Baylor to address health information queries is not always possible for caregivers due to family/work responsibilities, transportation costs and time constraints. The majority of caregivers are semi and low literate populations. No written material is used and thus there is a lack of reinforcement and support for remembering material learnt in the Baylor lectures. Most caregivers are uncomfortable with English, thus Baylor lectures and all interactions with caregivers are in Setswana. Baylor staff explains complex health information in accessible terms in local language.
Cure Answers to C om m on Questions about HIV Traditional Practices M yths Infection Learn about HIV How HIV works How HIV is spread? During pregnancy During Breastfeeding Preparing and Storing food Facts on Nutrition Food Groups

Although caregivers are encouraged to call Baylor with any questions they may have, most are reluctant (and unable) due to the high costs of mobile phone calls. These challenges and issues formed the basis of our motivation for the design of a health information SDS that provided not only adherence education from Baylor lectures but also general health information tailored to the needs of caring for HIV positive children. An SDS in the local language Setswana, toll-free and accessible at any time through a simple telephone call, could greatly support Baylor's services to caregivers of children with HIV. 2) Content Development Our design process started with the identification of relevant content and development of a framework (Fig.2) which detailed the broad health topics that needed to be covered by the SDS. Interviews with Baylor staff (nurses, doctors, nutritionists) and the following printed sources were used to create locally relevant accessible HIV health related content: Baylor Adherence lecture materials HIV Aids Care & Counselling, A Multidisciplinary Approach [19] Where there is no Doctor: A Village Care Handbook [20] HIV, Health & your Community, A Guide for Action [ 21]

Cereals/Rice/ Bread Fruits and Vegetables M eat & Beans M ilk and Dairy products Safe preparation of food Safe storage of food 1. H ygiene & Cleanliness

How H IV is spread

W hen to use gloves W hat to do if gloves are not available How to clean up blood properly. H ow to dispose infected gloves & waste W hen to wash your hands W ashing your hands with an antiseptic W ater shortage & hand washing Cleaning household item s How to wash your linen.

Dealing with body fluids & infected waste

The spread of HIV from M other to Child

2. Nutrition

Childcare 5. Facts about HIV 3. Com m on Sicknesses Diarrhoea Vom iting Breathing Problem s

G eneral Health H elpline
Danger Signs Hom e-based care Danger Signs Hom e-based care Danger Signs Hom e-based care Danger Signs W hen and How to give m edication ? W hat to do if you m iss an ARV dose? Traditional m edication Drugs & Alcohol 4. ARV M edication Learn about ARVs

W ashing your hands Keeping a clean household H ow AR Vs work ? M edication side effects How to store ARVs. W hen to give m edication? How to give m edication ? Planning ahead Vom ited dose Forgotten dose


Hom e-based care Danger Signs


Hom e-based care

Fig. 2. OpenPhone content framework.


We also held focus group sessions with 27 caregivers at Baylor on a second visit. The discussions were led by a Setswana facilitator and were aimed at soliciting input from caregivers on what kind of health information they typically require under the broad topics of the framework. The caregivers were first given a demonstration of a health SDS prototype and then asked what kind of information they would like the system to provide to them. For instance, the facilitator would introduce the topic of “Nutrition” and then ask e.g. “what kind of food does a child with HIV require? Caregivers’ responses were used to fuel discussion and direct further questions on sub-topics of the framework. Thereafter caregivers were asked to prioritize (rank) topics they had listed in order of need in a group consensus setting. In many cases caregivers debated and gave reasons for their choices. We found that the topics in our content were mostly inline with what caregivers proposed in the focus groups. However, the topics’ ranking done by the caregivers varied from ours, for example, they ranked Hygiene & Cleanliness as most important which we had thought was secondary to ARV medication. The rankings allowed us to determine the topics the SDS should focus on providing first and foremost and were later also used to determine to order of main menu options in the SDS. The focus group sessions allowed us to revise the content and framework to build a SDS that caters for the health information needs expressed by the caregivers themselves. 3) System Design Once the initial content had been created and approved, the dialogue flow and audio output of the SDS had to be designed in a user-friendly manner. We carefully crafted the primarily text-based content for spoken speech and simple language. For instance, words like “lower cranium” will have little meaning for semi & low literate users; such references were translated into more accessible language, using terminology and metaphors used by Baylor staff in their lecture sessions. For example, the white blood cells are the ‘soldiers’ of the body, and ARV medication is the ‘ammunition’ for these soldiers. Since the SDS was aimed at low literacy users and users who may never have used a SDS before, the design goal was to create an easy-to-use interface that placed a low cognitive load on the user. We grouped content under logical menus and sub-menus, so that the amount of audio output under each node was balanced with the number of menu options at any given point in time. The design team set the following constraints based on guidelines for speech user interface design [22, 23]: The maximum depth of menus in the SDS should not be more than 3 levels. The maximum breath or number of options at any menu should be no more than 5.

A sample system-user interaction follows. User: [Dials number...] System (Introduction): Hello and Welcome to the Health Helpline, I am Nurse Lerato and I know you probably have many questions about caring for someone with HIV. System (Overview): I can tell you about Hygiene & Cleanliness, Nutrition, Common Sicknesses, ARV Medication, and Facts about HIV. If at any time during your call you want to start our conversation again, you can press 0. System (Main Menu): For Hygiene & Cleanliness, please press 1, for Nutrition, press 2, for Common Sicknesses, press 3, for ARV medication, press 4 or for Facts about HIV, please press 5. User: [Presses 2.] System: Eating a balanced diet of different foods helps people with HIV stay healthy and strong. A healthy diet does not have to be costly and contains food from all the different food groups. Healthy food is always prepared and stored in a clean environment... As a final step, the design team decided on the specific wording of all audio prompts in Setswana. This required careful consideration for the dialect, the register (informal or formal), the cognitive load (short audio prompts, but long enough to provide contextual “anchoring”), and use of appropriate user interface metaphors. For instance, a system that suggests to the user “say Main Menu” is unlikely to make sense to users who have never interacted with visual or audio interfaces before. Instead, we chose the metaphor of a “conversation” and asked the user to “Start our conversation again”. The Setswana-speaking linguist on the design team played an essential role in the selection of ASR keywords that were locally relevant and logical to users, yet acoustically dissimilar. 4) Final SDS Development All the prompts and the health content were translated using a registered translation service into the dialect of Setswana spoken in Botswana. We specified that the content would be spoken aloud and the intended audience would likely be lowliterate. Since user interaction with an SDS is based on audio modality, the voice of the system plays a crucial role. With our target audience we felt that this was an essential element in creating a persona that would not only make users comfortable in their interactions with the system but also make the user experience enjoyable. Thus, our ideal voice talent would: Sound like a caring nurse willing to answer questions. Be a mother-tongue Setswana speaker. Have a full, mature female, well-articulated voice. Instil a sense of confidence and trust.


Our recruited voice talent was a well-regarded local soap celebrity, which meant her voice would be familiar to many of the target users. All the system prompts and content recordings were done in a professional recording studio. The SDS was built using the Asterisk telephony platform, the current set up in Botswana is a free standing PC, connected through an Asterisk card to an ISDN line. The ISDN line allows up to two calls simultaneously and users dial a local telephone number to access the system. While the system is operational, all aspects of the calls are logged. From start to finish all key-presses are monitored and all audio is recorded. C. User Study The goal of the study was to compare the most basic variant of speech systems (key-press replacement) with DTMF input. Thus, we built two identical systems that differed only at the menu prompts in choice of input modality i.e. in one system the user would press a key to chose a menu option and in the other they would say a keyword or phrase. For example, a DTMF menu option would be; “to hear about Nutrition, press 1,” whereas the ASR menu option would say, “to hear about Nutrition, say Nutrition.” The ASR was simulated using the Wizard-of-Oz (WOZ) methodology [24] where a researcher played the role of the speech recogniser. The ‘wizard’ listened to the speech input of the user and chose the next state of the system on this basis. The ‘wizard’ only accepted the exact keyword or phrase that the user was allowed to say at a particular menu option, any other input was directed to a No-Match state. In the case of the DTMF system, key presses were handled by the back-end telephony platform. Both systems ran from a PC laptop which was connected to standard telephone through a voice over internet protocol (VOIP) gateway.

due to either user’s time constraints or a technology failure experienced in system set-up. The experimental set-up included a facilitator, an observer and a WOZ operator. The facilitators were local graduate students who were trained by the authors to facilitate in the local language, Setswana. The observers took notes on user behaviour, number of verbal prompts needed by a user, any user comments, and general body language of the user. Each user was introduced to the system and asked to sign a consent form. Emphasis was placed on communicating that users were not being tested but rather the system for purposes of improvement. Thereafter, each user watched a five minute video showing how a caregiver could use the system to find typical health information that they might need (Fig. 4).

Fig. 4. User watching a full-context video.

The context of the video was carefully matched to depict typical scenarios where a caregiver could use the SDS to obtain health information. For example, in the first part of the video, a caregiver with a sick child who has just thrown up his ARV medication is unsure of whether to give the medication again. A friend of the caregiver arrives and tells her about a telephone information system that she can call and learn when and how the next dose of ARV medication can be given if a child throws up his medication. Research [25, 8] on designing interfaces for low-literacy users has shown that such a fullcontext video greatly improves user studies by offering not only a demonstration of how to use the interface, but also the source (and therefore trustworthiness) of the content, the context in which you might use such a service, and the potential impact in your day-to-day life. After this video, the facilitator gave a short demonstration to the caregiver on how to retrieve “Nutrition” information as shown in the second part of the video. For each task, the caregiver was asked to dial a number themselves to access the SDS. This simple action served as a quick check to verify the caregiver’s ability to recognise numbers and use a phone. In order to show the difference between the DTMF and ASR systems we used separate telephone sets for each system (Fig.

Fig. 3. DTMF vs. ASR experiment set-up.

The user study was held over a period of five days in April 2008 at the Baylor premises. A total of 33 caregivers were part of the study, of which 27 tried both the DTMF and ASR systems. The remaining 6 caregivers did not try both systems


3). After the caregiver finished the tasks using the DTMF system, for example, the facilitator explained that they would now try a similar system (on the other handset) but one where they would now say a keyword to obtain information. An example of a task explanation is shown on the following page. Facilitator: What’s the name of any good friend in your neighbourhood? Caregiver: Kabelo Facilitator: Your friend Kabelo says she must wash her hands frequently to keep her family safe from disease but she has very little water at home so she doesn’t know what to do. You’ve heard that Baylor has a phone number (help line) that can answer many health questions. So you decide to call the phone number, can you help your friend? Since the study was a within-subject comparison, we refrained from using the same tasks in both DTMF and ASR to prevent bias of previous knowledge. Thus, we created two sets of tasks; Set A and Set B with two tasks each (an easy and a difficult task; Task 1 and 2). The tasks were designed to be similar across Sets and to require a user to make three correct menu selections (Menu levels 1, 2, 3) to reach the specified information. Recall that our HIV health helpline is only three menu levels deep. The correct paths for each task are shown in Table I.

in Fig. 5). For example, if a caregiver started with DTMF using Task Set A he/she would then proceed to do ASR with Task Set B (Quadrant 1). Each caregiver thus did a total of 4 tasks and users were approximately evenly assigned between Set A (15 caregivers) or Set B (18 caregivers). The permutations in the order of trials also help to counter the impact of subject fatigue and learning effects within the small sample size.

Fig. 5. Order of Trials and Task Sets in experiment.

Task Description
Task 1, Set A Find out how to wash hands during water shortage. Task 1, Set B

Menu Level 1

Menu Level 2

Menu Level 3
Water shortage and hand washing

After completion of each modality’s trial, a post questionnaire was administered verbally to the caregiver. It consisted of ten questions adapted from the PARADISE evaluation framework [26]. The facilitator recorded the response in a 5-point Likert scale format based on the strength of the user response. We then verbally interviewed the caregivers to gather demographic data on education levels, language, and occupation, telephone usage (mobile and landline). Caregivers were also interviewed on their familiarity with technology (computer, mobile phone, TV, radio, video/DVD machine) based on factors such as use, ownership, frequency of use, place of use, and reason for use. Data was also gathered on the number of children they take care of, how often they visit Baylor and how they usually resolve their information queries. At the end of each session caregivers were provided with small non-monetary incentives (juice, potato chips and fruit for the child and gloves & household disinfectant for the caregiver) to thank them for their participation.

Washing Hygiene your and cleanliness hands

What to do if Body Hygiene fluids & gloves are not and Find out how to protect cleanliness infected available hands when gloves are not waste available. Task 2, Set A Find out how to care for a child with fever. Task 2, Set B Find out how to care for a child with diarrhoea. Common Fever Sicknesses Home Care *

Common Diarrhoea Home Care * Sicknesses

* Home Care can only be reached after the user has heard the Danger Signs of the selected illness. For this reason, Task 2 is slightly more difficult than Task 1.

Additionally, to minimize any possible bias due to the order of trial of the DTMF and ASR systems or the use of a Task Set (A or B) with a particular modality, we systematically ensured that our data covered all possible combinations of Order of Modality and Task Set (illustrated as Quadrants 1-4


Fig. 7. Methods of Health Resolution by caregivers.

Fig. 6. A user taking part in the study.

III. RESULTS In this section we present results from our user study, including a description of our users (Section A), their task completion scores, other usability metrics from the study and a comparison of their performances on the DTMF and ASR parallel systems (Section B). A. User Profile During our user study, we had 33 caregivers who participated, of which 27 tried both DTMF and ASR systems. The caregivers were all female with the exception of one male caregiver. The age of our users ranged between 22 and 61 years old with the average age of 34 years. The average number of years of schooling amongst our users was 9 years but 2 users had 0 years of schooling. All of the users could read and write the local language Setswana and approximately 79% of them knew some English. In terms of occupation, 47% were unemployed and of the 53% who were employed, the majority were in low-income occupations such as cook, cleaner, house maid, hair dresser, or security guard. Caregivers reported that they visit Baylor between 1-3 times a month, with average travelling distance and time at 28 km and 1 hour respectively, some travelling from as far as 130 km, with average cost of travel at 18 Pulas (approx. $3 USD). The average waiting time at Baylor reported by the caregivers was 2.5 hrs, also travel time to Baylor ranges from 30 minutes to 3 hours; together, these represent a significant portion of a working day and are a substantial burden to the caregivers. Caregivers were also asked what they usually do if they have questions regarding the child’s health and when the last time was that they had such a question. One third of the caregivers usually go the local clinic to resolve their health queries and another quarter go to Baylor for this purpose (Fig. 7). Forty percent of the caregivers had a query regarding the child’s health within the last 6 months, and another 21% had more recent queries; in contrast, only 12 % could not remember specific queries (Fig. 8).

In terms of technology familiarity, 30 out of 33 (91%) caregivers owned mobile phones and 85% of these knew how to load their mobile phones with ‘airtime’ (pre-paid phones which require users to load money by calling the network provider’s service number and entering a sequence of digits from the pre-paid calling card). Average mobile phone costs per month were 68 Pulas ($10.5 USD) with an average cost per call being reported as 4.5 Pulas ($0.75 USD). Only 30% of caregivers reported having access to a landline telephone and of these only 9% had the landline at home (Table II).

Fig. 8. Time since last Health Question by caregivers.


Technology Mobile Phone Landline Computer TV Radio DVD/Video Machine ATM

Use 91% 30% 15% 76% 91% 41% 35%

Ownership 91% 9% 3% 71% 91% 38% N/A


B. Broad Usability Metrics and Observations System usability was determined using objective (Task completion rate, response time, routing time) and subjective (user preference, Likert ratings) metrics [27]. 1) Response Time Response Time measures the time it takes someone to respond to the system for the first time. It is usually measured not from the system start, but from the end of the Main Menu prompt, which in our case was 29 seconds long. This results in three basic categories of responses: (1) people who barge-in will have a negative response time, (2) people who respond in the 4 seconds of silence after the end of the Main Menu prompt will have a response time between 0 and 4 seconds, and (3) people who listen to one or several timeouts before responding will have a response time greater than 4 seconds. The Mean Response Times of our users (Fig. 9) indicates many barge-ins, roughly corresponding to the time the correct option at the first menu level was played: Hygiene and Cleanliness for Task 1, Common Sicknesses for Task 2 (Table I). Although we expected that our users would exhibit some short term learning effects, we found that response time did not decrease from the first call to subsequent calls.

Fig. 10. Task Completion Rates for DTMF and ASR.

For this study, we provide the task completion rate at each menu level of the SDS (Fig. 10). Across both tasks and both input systems, 60% to 84.38% of caregivers selected the correct first menu option (Menu Level 1). Approximately half of those people were able to select the appropriate sub-menu option (Menu level 2) when the task was to find out about “Common Illnesses” (Task 2), with the ASR system yielding the highest task completion rate for this menu level (56.67%). Caregivers seemed to have had more trouble correctly selecting Menu level 2 options for Task 1, with ASR again yielding the highest task completion rate of the two systems (40.63%). Task completion rate for level 3 ranged from 2.85 % to 23.68%, with Task 2 causing the most difficulty. Rates did not vary significantly by input mode (DTMF vs. ASR) or by task set (A vs. B). 3) Routing Time Routing Time measures the time it takes a user to reach the beginning of the node which contains the correct information for the assigned task. Fig. 11 shows the mean routing time from our user study. Users’ routing time was similar across both tasks and both systems for both Level 1 and Level 2. As expected, due to the forced loop through Danger Signs for Common Sickness, caregivers took longer at Level 3 for Task 2.

Fig. 9. Average Response Time (time from end of the Main Menu prompt until the user's first input).

2) Task Completion Rate Task completion rate measures how frequently users were able to reach the node that provided the correct information for their assigned task.


Fig. 11. Routing Time for completed tasks. Recall from Task Completion Rates (above) that the Level 1 mean response is averaged over approximately 75% of our users, whereas Level 2 and Level 3 times are averaged over fewer users, in some cases only one or two.

Fig. 12. Usability Metrics for DTMF and ASR systems.

5) User Preference Systems were rated separately using the PARADISE [26] framework after use. The Likert scores were found to be unreliable, however. Despite efforts to elicit honest, critical feedback to the system (for example, we had a different person in a separate room conduct the post-study evaluations), all caregivers gave the system the highest marks possible across all categories and were hesitant to provide any criticism. For those 27 (out of 33 total) caregivers who tried both the DTMF and the ASR system, we were able to elicit feelings of preference for one system over the other (Table III). Most caregivers (59%) preferred the DTMF system over the ASR one (19%) and 22% indicated no preference. Both the DTMF and the ASR systems were judged to be the faster system by those who preferred it. Our measurements of routing time and task completion, however, show the systems are comparable.

Only two users correctly completed Task 2 with the ASR system; both users took over 4 minutes to do so. With the DTMF system only one person completed Task 2, doing so in less than 3 minutes. All users that correctly completed Task 1 using the ASR system did so in about 60 seconds. For Task 1 on the DTMF system, most users completed the task in just over 60 seconds, but one user took over 6 minutes.

4) Other Usability Metrics Results from our user study show that user interaction with the ASR and DTMF system are very similar across a wide range of additional usability metrics, illustrated in Fig. 12. Use of the Main Menu global (press '0' or say 'Simolola') and the Exit global (press '9' or say "Fetsa"), were similar for both systems. There were almost the same average number of timeouts (when 4 seconds elapsed with no user response) and repeats (when user chooses to repeat an information node) for the ASR and the DTMF system. On average, caregivers used the barge-in function one more time when using the ASR system but the total number of turns taken by users was similar for both systems. Note, all the usability metrics are means for each call, whether the task was completed or not.

Preferred Num Reasons Given System ber DTMF 16 Clearer instructions (7) Faster to use (4) More private (2)

Example Remark "Its quick and the doctor gives you the instructions, you just have to follow them."



More accessible "Its faster and old people can also use (2) it. The button Faster (1) system takes too Clearer long. " instructions (1) Hands free (1) Similar (2) "They are similar. With one you press buttons, with the other, you say

Both None

or 6


things." 6) Social Factor Correlations We examined several social factors based on our users' responses to questionnaires to see if they correlated with either their interaction with the system, based on objective metrics, or their reported system preference (DTMF vs ASR). Employment and experience loading airtime on a mobile phone were significant factors in the overall task completion rate (p=0.09 and 0.02, respectively). Those users who were employed or who had experience loading airtime completed more tasks during the study. Age, education, use of landlines, mobile phones, or ATM machines were not significant factors. Only previous use of a landline was a significant factor in use of system globals (p=0.01). Use of landlines was also a factor, along with loading airtime, in whether a user barged-in during system use (p=0.1 and 0.1 respectively). Caregivers who use a landline and load airtime were more likely to barge-in during system use. No questionnaire responses were found to be significant factors for system repeats or overall response time. Employment, previous use of an ATM, and experience loading airtime on a mobile phone were significant factors in the overall correct response (p= 0.07, 0.09, and 0.03) 1 . Caregivers who were employed, used ATM machines and loaded airtime had more correct responses to tasks during the user study. Age, education, and amount of mobile phone use were not factors in correct response. Loading airtime was the sole significant factor in user preference of DTMF over the ASR system (p=0.1). Those people who load airtime regularly preferred DTMF over ASR. Employment significantly correlated (p<0.1) with Overall Task Completion and Overall Correct Response. Age and monthly mobile phone costs were not found to correlate significantly with user performance or user preference for either the DTMF or ASR system. 7) Other Observations We observed based on body language and explicit remarks that several caregivers were nervous at first, and then became more relaxed during the first few minutes of the study. Many caregivers (and their children) showed signs of fatigue while trying the second system (sometimes an hour later). Caregivers sometimes had trouble understanding the task. They would often try to find information about ‘Nutrition’, for example, which was the topic of the demonstration, instead of searching for information on their assigned task. Caregivers also often clarified with the interpreter what the task was, what the keywords were, and what they should press or say during their trial. The interpreter would nod or say 'yes' but
The following categorical groups (yes and no) were divided into two groups and the resulting dependent values analysed: Employment, previous use of an ATM, experience loading airtime, use of landlines, use of ATM machines. A p-value of 0.1 and less indicates a significant difference between the two groups, whereas a p-value higher than 0.1 indicate no significant difference.

would not help the caregiver any further. In some cases, they would ask to try again, which we allowed if time permitted. Most were very interested in the content and many referred to the voice they heard as ‘the doctor’. Only one caregiver recognized the voice of the SDS as the celebrity soap star although most commented that “the ‘doctor’ explained very nicely”. During the interviews, all of our users enthusiastically indicated that they would like to use the service again; many said that it would be very valuable for educating themselves and their family/friends on caregiving aspects for children with HIV. A SDS such as OpenPhone could also serve as persuasion tool for caregivers trying to educate others, as explicitly reaffirmed by a caregiver “now I can tell them at home that the doctor (SDS voice) says the same thing (referring to a HIV related topic) that I’m telling them”. IV. DISCUSSION From our pilot study, we found that there were no significant differences between task completion rates (ASR only performed slightly better) or other usability metrics for both systems. This agrees with a number of previous studies in the developed world [12, 28, 29] where no major differences were found in terms of performance. However, subjectively the majority of our users preferred DTMF (59%) over ASR (19%), which is in contrast to formal studies in the developed world [9]-[11] (where user preferences largely favour speech), but correlates with the observation that simple key-press replacement with keywords is generally not viewed favourably. The users who did prefer ASR did not as in developed world studies comment on the aesthetics of speech input, that “speech is more entertaining or enjoyable” but rather on the utility of speech “more accessible for older people or faster”. Our finding that users’ employment and experience loading airtime correlated with higher task completion but that education level does not, indicates that technological literacy is a more important factor in adopting new technology than literacy itself. This may also contribute to the finding that ‘loading of airtime’ was the sole significant factor found in DTMF preference over ASR. It is also interesting to note that our users who had minimal exposure to SDSs (except loading airtime), were relatively comfortable using our system for the first time, as indicated by their frequent, timely barge-ins. Also, our users noticed the value of speech (allowing hands-free operation, innumerate/older people being able to use it) and DTMF as well (provides privacy). This highlights that even if users may be technically inexperienced and unfamiliar with an ICT application, they have valuable and sound judgment on the utility of the interface. Also, both DTMF and the ASR systems were judged (subjectively) to be the faster system by those who preferred it, which indicates that time constraints, may also be of importance for low literacy users, in contrast to


earlier observations. The DTMF vs. speech input comparison could be improved in terms determining task performance by using a between-subjects experiment design; however this approach would not reveal user preference. Our preliminary investigations indicated that a good fraction of caregivers were in the low and non–literate range. In our user study though, we encountered that many of our recruits were semi- to low literate. Also, whilst our user numbers represent a significant sample size for a developing world user study, it may in comparison to studies in the developed world be on the ‘small’ end of the spectrum. The above-mentioned issues emphasize some of the challenges faced in research for the developing world; that the very users for whom an SDS could be most useful may be the hardest for us to reach and also that user recruitment in developing regions can present significant obstacles [30]. Whilst this pilot study illustrated that telephony services could in fact be easily used by semi and low literacy users, and that an SDS in local language can be a powerful health education tool, the decisive factor in widespread uptake is likely to be the cost incurred by the caller for the service. The majority of caregivers said that even though the service would be useful to them they would only be able to make use of it, if the service is toll-free. An average phone call in Botswana of 5-10 minutes to the SDS would cost a mobile phone user $1-2 USD. From our questionnaires, the average cost per month for mobile phone usage was $10.5 USD. Thus, a single phone call to the SDS would consume, 10-20% of a caregiver’s monthly mobile phone budget, making the case for a toll-free number all the more imperative. Notwithstanding our findings that show promise for SDSs for low literacy users, we did encounter challenges in introducing the concept of a SDS to our users, for instance a few users did not fully realise that the system was automated. For example, at the end of one call, a user proceeded to ask the ‘nurse’ (system persona) a question when prompted by the system to leave a comment (and waited for the answer). Another user repeatedly acknowledged what the ‘nurse’ was saying by responding with “Yes, yes” or “I agree with you”. An obvious solution here might be to use a text-to-speech (TTS) voice for the prompts. However, we run the risk of a mismatch between the target audience and the context (e.g. emotionally sensitive in terms of health care) or culturallybased communication norms of the community. This in turn may affect the willingness of users to interact with the system. A general design challenge for SDSs is to ensure a minimal cognitive load on the user. This magnifies more so in the case of information access applications where lengthy pieces of information need to be provided (e.g. in a health care context). In our case, all the users indicated in the subjective questionnaire that the length of the content was not too long; however, during the experiment some users did mention the need to concentrate in order “not to miss all the things being said by the ‘nurse’ ”. There is a need to address the abovementioned issues by developing the dialog to be more

interactive (perhaps with audio cues, getting intermittent user feedback) and conversational. In addition, the nature of educational information services tends to be rather exploratory; where a user may peruse various topics related to his/her general query (e.g. a user may want to, in general know about “Dealing with body fluids and infected waste” which has 4 topics all related to that option). This in our case translated to some users struggling to find the exact menu option related to the task and exploring related sub-topics. One user even singled out that she would like to know the mapping of the menu options beforehand to help her locate the information she is looking for. This experience highlighted challenges of using hierarchical menus [8] and the importance of paying attention to the taxonomy and vocabulary of the system to enable easy navigation for the user. SDS design for smaller languages also introduces challenges on other dimensions including prompting and persona. The prompts and content of a SDS application will typically be translated from a language such as English to the local language. Thus, great care has to taken in the prompt writing phase to ensure that intended meaning of the original prompt (English) is still preserved in the translated prompt (local language) and conveyed in the simplest and shortest way possible. Often, a concept described by a single word in English has no direct translation in another language. For instance, whereas a keyword in the English version of our application was “Safe food”, it became the phrase, “Dijo tse di siameng” after translation, in order to adequately describe the concept. Moreover, not only should the translated SDS prompts convey the intended meaning but the designer should ensure that the persona of the local language system is in line with cultural and contextual expectations of the intended audience. For instance we ensured our prompts not only had the right balance of formality and gravity appropriate for the message (HIV info) but were also understandable and conversational. Our experience in employing multiple data-gathering techniques in the needs investigation phase (interviews and discussions, observations, field visits, and focus groups) better equipped us in trying to comprehend the needs of our users. The interviews and discussions helped us establish a rapport with our users and stakeholders (Baylor staff), whilst also providing the flexibility of follow-up questioning and recalibration of interviewer terminology when needed. Observations on the other hand helped to further reveal the issues that users are unable or unwilling to articulate or express. Field work allowed us to pick up on the cultural nuances and social-economic context of the users and enabled us to gain a deeper understanding of the sensitive environment (HIV/health) we were working in. Finally, through focus groups we were able to observe the interaction amongst caregivers and most importantly it enabled us to obtain specific targeted design information in a group setting and correct our earlier assumptions on the ranking of SDS health topics.


REFERENCES V. CONCLUSION This paper has addressed the frequently debated question of input modalities of touchtone vs. speech in a completely new context; low literacy users and a domain (health) different from the usual business centric applications (call routing, voice mail systems or banking). Our pilot study also served to confirm the feasibility of SDS applications for semi and low literate populations. It is interesting to speculate on the applicability of our findings to other resource-poor countries. Our intuitive judgment is that the only characteristic of our user population that strongly influenced our results was their relative familiarity with mobile telephones; however, the matter requires detailed investigation. In future work we would therefore like to further explore the space of non-literate users and the suitability of SDS applications for them, as well as investigate the interaction of task types (linear vs. non-linear) and application domain – informational (Openphone) vs. transactional (tracking a social services payment) with the input modality. Once the system is free of charge and has been in use for several months, we would like to study whether OpenPhone leads to a change in health habits and improves the ability of a caregiver in providing care to children, like change of hygiene habits, better nutrition, and fewer misconceptions about HIV. Throughout the design and development process, we experienced that beyond usability and creating simple, accessible user interfaces for low literacy users, factors such as cultural and social context, establishing relationships with stakeholders and user communities, and localizing content, play a vital role in the success of an ICT intervention in the developing world. We intend to draw on these valuable experiences and carry our work forward in serving the information needs of citizens of developing regions with accessible and usable telephony based services. ACKNOWLEDGEMENTS The authors wish to thank the staff of Botswana Baylor Children’s Clinical Centre of Excellence for the generous manner in which we were received, in particular Dr Paul Mullan, for the amount of help and time given to us. We would also like to thank members of the HLT group at Meraka Institute who provided valuable contributions throughout the project; Victor Zimu, Jama Ndwe, Mpho Kgampe, Louis Joubert, Richard Carlson, Bryan Mcalister, Mark Zsilavecz, Marelie Davel and Alta de Waal. We also greatly appreciate the time given to us by Connie Ferguson, our celebrity voice talent. This research would not have been possible without the joint support of OSI and OSISA. The project was also in part supported by the NRF under the Key International Research Capacity (KISC) programme, UID no. 63676.
[1] [2] [3] R. Tucker and K. Shalonova. "The Local Language Speech Technology Initiative", SCALLA Conference, Nepal, 2004. SIL International. (2001). Facts about illiteracy. [Online] Available at:: http://www.sil.org/literacy/litfacts.htm (last accessed 15 Sept 2008). ITU: International Telecommunications Union. (2003). Mobile overtakes fixed: Implications for policy and regulation. [Online]. Available: http://www.itu.int/osg/spu/ni/mobileovertakes/Resources/Mobileovertak es_Paper.pdf, (last accessed 5 September 2008). Statistics South Africa. (2007). Community Survey 2007. [Online] Available at: http://www.statssa.gov.za/publications/P0301/P0301.pdf (last accessed 15 Sept 2008). E. Barnard, L. Cloete, and H. Patel. “Language and Technology Literacy Barriers to Accessing Government Services,” Lecture Notes in Computer Science, vol. 2739, pp. 37-42, 2003. M. Plauche, U. Nallasamy, J. Pal, C. Wooters, and D. Ramachandran. “Speech Recognition for Illiterate Access to Information and Technology,” in Proc. IEEE International Conference on Information and Communications Technologies and Development ‘06, pp. 83-92, May 2006. P. Nasfors. "Efficient Voice Information Services for Developing Countries", Master Thesis, Department of Information technology, Uppsala University, Sweden, 2007. J. Sherwani, N Ali, S. Mirza, A. Fatma, Y. Memon, M. Karim, R. TRongia and R. Rosenfeld, “Healthline: Speech-based Access to Health Information by low-literate users”, in Proc. IEEE International Conference on Information and Communications Technologies and Development ‘07, Bangalore, India, Dec. 2007. S. Agarwal, A Kumar, AA Nanavati and N. Rajput, "VoiKiosk: Increasing Reachability of Kiosks in Developing Regions", in Proc. of the 17th international conference on World Wide Web, pp. 1123-1124, 2008. K. M. Lee and J. Lai “Speech Versus Touch: A Comparative Study of the Use of Speech and DTMF Keypad for Navigation” International Journal of Human–Computer Interaction, vol. 19, no. 3, pp.343–360, 2005. B. Suhm, J. Bers, D. McCarthy, B. Freeman, D Getty, K Godfrey and P Peterson, "A comparative study of speech in the call center: natural language call routing vs. touch-tone menus". In: Terveen, Loren (ed.) Proc. of the ACM CHI 2002 Conference on Human Factors in Computing Systems Conference, Minneapolis, Minnesota. pp. 283-290, April 2002. C. Delogu, A. Di Carlo, P Rotundi and D. Sartori, "Usability evaluation of IVR systems with DTMF and ASR", in Proc. of the 5th International Conference on Spoken Language Processing (ICSLP 98), paper 0320, Australia,1998. T. Pearce and M Bergelson. (2008). Alignment index for speech selfservice. Dimension Data Technical Report [online]. Available at http://www.dimensiondata.com/NR/rdonlyres/9191A848-5F35-459F82398D9D2248414E/8791/mainstreamspeechalignmentindexreport2.pdf (Last accessed 15 Sept 2008). J.P. Migneault, R. Farzanfar, J.A. Wright, and R.H. Friedman. “How to write Health Dialog for a Talking Computer,” Journal of Biomedical Informatics, vol. 39, no. 5, pp. 468 – 481, Oct. 2006. Tellme Networks Inc. Who uses TellMe? [Online]. Available at: http://www.tellme.com/about (Last accessed 15 Sept 2008). UNAIDS (August, 2008). Report on the Global AIDS Epidemic. Joint United Nations Programme on HIV/AIDS. [Online] Available at: http://www.unaids.org/en/KnowledgeCentre/HIVData/GlobalReport/200 8/2008_Global_report.asp (last accessed 7 Sep 2008). K. Seipone, W. Jimbo, F.d.l.H. Gomez, K. Ampomah, J. Othwolo, O. Kaluwa, M. Busisiwe. “Trends in HIV Prevalence Among Pregnant Women in Botswana 2001-2005” 16th Intl. Conf Aids. Toronto, Canada, Aug. 2006. Botswana-Baylor Children’s Clinical Centre of Excellence, Annual Report 2006, Gaborone, Botswana, 2006. A. Van Dyk, HIV Aids Care & Counselling, a Multidisciplinary Approach. 2005, Third Edition: Pearson Education, South Africa.

[4] [5] [6]

[7] [8]






[14] [15] [16]


[18] [19]

[20] D. Werner, C. Thuman and J. Maxwell. Where there is no Doctor, a Village Care Handbook.. 2007, New revised edition: Hesperian, CA, USA. [21] R. Granich and J. Mermin. HIV, Health & your Community, A Guide for Action. 2001, Hesperian, CA, USA. [22] G.A. Miller. "The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information," The Psychological Review. vol. 63, no. 2 pp. 81-97, 1956. [23] Suhm B. “IVR Usability Engineering using Guidelines and Analyses of end-to-end calls” in D. Gardener-Bonneau and H.E. Blanchard (Eds). Human Factors and Voice Interactive Systems. 2008, pp. 1-41, Second Edition, Springer Science: NY, USA. [24] N.M. Fraser and G.N. Gilbert, “Simulating speech systems”, Computer Speech and Language, vol. 5, no. 2, pp. 81-99, 1991. [25] I. Medhi and K. Toyama, “Full-Context Videos for First-Time, NonLiterate PC Users” ”, in Proc. IEEE International Conference on Information and Communications Technologies and Development ‘07, Bangalore, India, Dec. 2007. [26] M.A. Walker, D. Litman, C.A. Kamm and A. Abella. “PARADISE: A general framework for evaluating spoken dialogue agents” in Proc. of the 35th Annual Meeting of the Association of Computational Linguistics. ACL/EACL 97, 1997. [27] Nielsen, J., (1993). Usability Engineering. AP Professional, Boston, MA, USA. [28] Foster, J. C., McInnes, F. R., Jack, M. A., Love, S., Dutton, R. T., Nairn, I. A., et al. (1998). An experimental evaluation of preference for data entry method in automated telephone services. Behaviour & Information Technology, 17, 82–92. [29] Goldstein, M., Bretan, I., Sallnas, E.-L.,&Bjork, H. (1999). Navigational abilities in voice-controlled dialogue structures. Behaviour & Information Technology, 18, 83–95. [30] E. Brewer, M. Demmer, M. Ho, R.J. Honicky, J. Pal, M. Plauche, and S. Surana. “The Challenges of Technology Research for Developing Regions,” IEEE Pervasive Computing, vol. 5, no. 2, pp. 15-23, AprilJune 2006.


ICT4What? – Using the Choice Framework to operationalise the Capability Approach to Development
Dorothea Kleine
development outcomes top-down and a-priori is unsuitable in the context of multi-purpose technologies which could empower individuals to attain development outcomes of their own choosing. The paper is structured in six parts. In section one I briefly introduce Sen’s capability approach before presenting, in section two, some important steps towards operationalising it. Building on this body of work, section three develops the Choice Framework as a further way of operationalising Sen’s approach. Methodological implications are explored in section four before in section five the Choice Framework is applied in a case study of one particular individual’s usage of the Internet in a telecentre in rural Chile. The final part points out limitations of the model, directions for further empirical research and calls for more theoretical work on the nature of the “development” element in ICT4D. The paper concludes by highlighting some concrete implications this theoretical work may have for practitioners. II. EVOLVING THEORIES OF DEVELOPMENT Research located in the contested intellectual space that is ‘development’ needs to be able to answer the fundamental question of what is understood as development. Broadly speaking, debates in development studies range from positions which equate development with economic growth (e.g. [2], [3], [4], [5]) through to critical perspectives stressing that uneven development, dependency and inequality are inherent in capitalist development (e.g. [6], [7], [8], [9]) to ideas of alternative, bottom-up development recognising social and ecological as well as economic goals (e.g. [10], [11]), and radical “post-developmentalist” critiques that often dismiss the entire “development project” altogether (e.g. [12], [13]). The most influential challenge to the mainstream growthfocused view of development has come from Amartya Sen’s capability approach (also known as the capabilities approach) in which development is defined as “a process of expanding the real freedoms that people enjoy to lead the lives they have reason to value” [14]. His understanding focuses on development as freedom of choice. While this understanding of development is a minority position within institutions such

Abstract— Identifying the specific contribution of the use of ICTs to specific development goals has proven to be extremely difficult. This paper argues that instead of trying to make ICTs fit with a linear conceptualisation of impacts and an often economistic view of development, ICT4D should be used as a prime example of a development process which has to be analysed in a systemic and holistic way. Amartya Sen’s capability approach offers a way of thinking about development not as economic growth, but as individual freedom. The Choice Framework is presented as a way of operationalising this approach and visualising the elements of a systemic conceptualisation of the development process. An individual case study, related to telecentres in rural Chile, is used to demonstrate the way the Choice Framework can be applied as a guide to a systemic and holistic analysis. Index Terms— ICT4D, Amartya Sen, capability approach, Choice Framework, telecentres, Chile

he paradox is this: ICTs and particularly the internet are widely regarded as groundbreaking inventions that have changed the way millions of people live their lives, and yet researchers and practitioners in the field of ICT and development often struggle to prove specific impacts of the technology to funders. There may be specific reasons why particular projects fail, even some generalisable patterns of failure [1], but the overall degree to which the ICT4D community has to struggle when trying to legitimise its work to funders is astonishing in the context of a general discourse about how much these technologies have changed our lives. This paper tries to unravel the reasons behind this paradox by arguing two fundamental points: On a theoretical level, while there have been interesting alternative theoretical approaches to development, including Amartya Sen’s capability approach, the mainstream discourse’s conceptualisation remains heavily focused on economic growth, which is too narrow to capture the impacts of ICT. Secondly, and on a practical level, the common way of measuring impact by defining the intended
Manuscript received September 22, 2008. This work was supported in part by the Dr. Heinz Dürr Fellowship Programme. Author’s Details: Dorothea Kleine is Lecturer in Human Geography at Royal Holloway, University of London, Egham, TW20 0EX and a member of the UNESCO Chair/ICT4D Collective (email: dorothea.kleine@rhul.ac.uk).




as the World Bank [15], Sen’s approach has influenced the development discourse and it provides a means of building a bridge between those working in international development organisations and researchers in academia. The approach developed by Amartya Sen ([16], [17], [18], [19]) argues that development is about the freedom of choice in the personal, the social, the economic and the political sphere. In Sen’s approach, “functionings” are the various things a person may value doing or being, such as being adequately nourished, being healthy and being able to take part in the life of a community. In Sen’s terminology a person’s “capability” refers to the alternative combinations of functionings that are feasible for her/him to achieve [20]. The focus of development thus becomes increasing a person’s capability set, or her/his substantive freedom to lead the life she/he values. Functionings represent the “outcome” component, while capabilities are the “freedom” component in this approach. 1 In Sen’s more holistic view of development, economic growth plays an important, but not exclusive, role. Sen suggests ontologically focusing on human wellbeing and methodologically focusing on capabilities. While welcoming Sen’s approach at offering a more holistic view of development, scholars have been struggling to find a balance between its conceptual richness and its potential to be operationalised. Several scholars [21], [22], [23], [24] have attempted to operationalise the approach. Within this field the majority of studies use capabilities as a normative basis for the research while measuring functionings as a proxy [25], owing to the practical difficulty of measuring capabilities. Some authors (e.g. [26], [27]) have challenged Sen to draw up a general list of capabilities, but Sen has refused to do so, claiming that specific lists of capabilities ought to be drawn up for a given research or policy context [28] and, crucially, that the process of choosing capabilities should be left to the individual [29]. The dilemma which emerges is how to apply the capability approach to specific areas or sectors in a meaningful way while retaining openended development outcomes that do not presuppose individuals’ choices. II. OPERATIONALISING SEN’S APPROACH Sen intended his approach to be combined with other theoretical approaches [30]. The following section explains how, in order to operationalise the approach for ICT4D and other areas of development, elements have been drawn together from the literature on empowerment and on sustainable livelihoods to enhance the application of Sen’s approach.

A. Empowerment One of the most interesting attempts to operationalise Sen’s ideas is offered by Alsop and Heinsohn [31]. Writing for the World Bank, they link choice with their definition of empowerment 2 . They define empowerment as “enhancing an individual’s or group’s capacity to make effective choices and translate these choices into desired actions and outcomes” [39]. ICTs could be seen as useful tools in such processes of empowerment Alsop and Heinsohn see material and non-material assets, or resources, as the basis of individual agency which, together with the structural conditions frame empowerment processes. In their attempt to use empowerment as a middle-range theoretical concept to convert the development paradigm of choice into a construct that is of use to practitioners, Alsop and Heinsohn build a crude framework which connects “individual agency” with an “opportunity structure” from which follow the degree of empowerment an individual has to achieve development outcomes. The different “degrees of empowerment” are: existence of choice, use of choice and achievement of choice [40]. Individual agency is measured by an individual’s asset endowment, consisting of “psychological, informational, organisational, material, social, financial or human” assets [41]. These assets are listed, but not defined. An actor’s opportunity structure is said to be shaped by the “presence and operation of the formal and informal institutions” [42] and measured by the presence and operation of laws, social norms and customs. Alsop and Heinsohn have applied their framework in the evaluation of World Bank projects with women, on rural water supply and sanitation, on school decentralisation and with school dropouts. B. The sustainable livelihood framework Another literature which can be linked to the capability approach is the literature on livelihoods. Based on earlier work on livelihoods ([43], [44], [45] the Sustainable Livelihood Framework (SLF) used by the UK Department for International Development (DFID) [46] offers an analytical tool to understand in a systemic way the elements influencing the lives of the poor. Duncombe has demonstrated how the SLF can be applied to ICT4D research with microenterprises [47], while retaining the focus on poverty reduction through economic growth. The SLF includes the concept of an individual’s “capital portfolio” made up of five “capitals”: human capital, natural capital, financial capital, physical capital and social capital. In operationalising the SLF, human capital is measured by formal education and health indicators, but there has been a
The concept of empowerment originated in work on gender relations and community participation (e.g. [33], [34]) and has been increasingly discussed in development studies (e.g. [35],[36], [37], [38]. There are several competing definitions of the term.


For a more in-depth discussion of the capability approach, see also [32].


struggle to quantify “social capital” [48]. As a result, critics have argued that “everything social” gets packed into the social capital variable [49]. Individuals own or have access to this portfolio of capitals, their “livelihood assets” with which they negotiate “policies, institutions and processes”. They operate within a “vulnerability context” and develop livelihood strategies which then result in livelihood outcomes. Livelihood outcomes are defined a priori – In the DFID version, “more income” is listed at the top, even before “increased wellbeing”. The SLF offers a broad and systemic view of development processes, but its

A. Outcomes True to Sen’s statement that choice is both the aim and the principal means of development [51], the primary development outcome is choice itself. Secondary development outcomes depend on the individual’s choice as to what lives they value. These may include, for example, easier communication, increased knowledge, more income or time saved. Information and communication technologies might prove useful tools in achieving these outcomes. Just like other attempts to operationalise Sen’s work, here capabilities are not measured directly, though participatory research with individuals and groups may reveal them to some degree. Mainly, the outcome component will map or measure the

• • • • institutions and organisations discourses policies and programmes formal and informal laws including: - norms on usage of space - norms on usage of time • access to ICTs - availability of ICTs - affordability of ICTs - necessary skills for ICTs
Key: ER = Educational Resources PsR = Psychological Resources In = Information FR = Financial Resources CR = Cultural Resources SR = Social Resources NR = Natural Resources MR = Material Resources GR = Geographical Resources He = Health

DEVELOPMENT OUTCOMES DIMENSIONS OF CHOICE • existence of choice • sense of choice
Principal: Choice

Secondary: • easier communication • increased knowledge • access to markets • business ideas CR
Age Gender Ethnicity


• use of choice • achievement of choice

• increased income • more voice • time saved • higher job satisfaction




Fig. 1: The Choice Framework set of capitals is limited and in it the development goals are predetermined and not up to the individual to choose. In this respect, the SLF fails to mirror the thinking behind Sen’s approach. III. THE CHOICE FRAMEWORK Based on Sen’s capability approach, inspired by Alsop and Heinsohn’s work on operationalising Sen’s work, taking elements from the SLF and informed by an in-depth research project with microentrepreneurs’ use of ICTs in Chile [50], the Choice Framework was developed. After presenting it in diagrammatic form (Fig 1), the following sections will in turn explain each of the key components of the framework. achieved functionings resulting from an individual’s choices as a proxy to the capabilities. 3 An analysis based on the Choice Framework would then work backwards, from the outcomes, into the systemic relationships between agency, structure and choice, thus analysing how the outcomes were arrived at. B. Dimensions of Choice Alsop and Heinsohn’s dimensions of choice, which they call “degrees of empowerment” include, firstly, the existence
3 Two disadvantages of this method are that some of the individual’s capabilities are not captured in the achieved functionings and that it is difficult to trace the choices related to apparently negative outcomes. However, so far, capturing functionings is methodologically easier and more precise – in regards to both quantitative and qualitative methods - than capturing capabilities.


of choice – whether the different possibilities exist and are, in principle, attainable for the individual if the combination of their resource portfolio and the structural conditions would allow it. The second dimension, a sense of choice, not originally included by Alsop & Heinsohn, was added as a result of fieldwork experiences relating to ICT and development. Individuals were aware of some possibilities the new technology offered them, like email and online chat, but not of others, like Voice over IP. This was precisely because their educational resources (including computer skills) and the dominant discourse in the Chilean media stressed some usages over others. For any piece of research focused on a technology which is new to the respondents, the dimension of “sense of choice” will play a significant role. The “use of choice” dimension refers to whether or not an individual actually makes the choice and the “achievement of choice” refers to whether the outcome matches the choice expressed.

face to face contact in the urban economy [55]). Human Resources: The term “human resources” has been used for decades in the economics and industrial relations literature. 4 In the Choice Framework, this term needs to be disaggregated into Health and Education and Skills (educational resources). Within Sen’s paradigm of development, good health is a prerequisite for a person’s ability to choose the life she/he values. Educational resources represent education and skills acquired through formal and informal means. Psychological resources: Alsop and Heinsohn [56] recognise the significance of “psychological assets” and give as an example “capacity to envision”. More broadly, psychological assets may include self-confidence, tenacity, optimism, creativity and resilience. Spiritual or religious beliefs stand in complex interrelation with psychological resources – they can strengthen or weaken an individual’s psychological resources. Information: Alsop and Heinsohn list informational assets as a key resource. Heeks [58] calls for putting information at the centre for analysis of ICTs and Development, and Gigler [59], adds “informational capital” to the capital portfolio. Access to information is the first step to knowledge acquisition, the process of filtering and transforming information into meaningful knowledge. Cultural resources: “Cultural capital” – which in the Choice Framework is called cultural resources – exists, according to Bourdieu [60], in three states: an embodied state (the habitus a particular person lives in); an objectified state (objects like paintings, instruments and monuments which only the initiated can use or appreciate); and an institutionalised state (prestige attached to, for example, academic titles). Social resources: “Social capital” – or social resources – is included in both the SLF and Alsop and Heinsohn’s work. It has been both immensely influential and highly contested in development discourse. For the Choice Framework, Bourdieu’s definition of social capital is used: “the aggregate of the actual and potential resources which are linked to possession of a durable network of more or less institutionalized relationships of mutual acquaintance and recognition – or in other words, to membership in a group – which provides each of its members with the backing of the collectivity-owned capital, a “credential” which entitles them to credit, in the various senses of the word.”[61] Membership of these groups can be defined by kinship, friendship, shared ethnicity or class, friendship or informal commonality ties.

C. Agency Instead of using a terminology of capitals and asset or capital portfolios, Sen uses the term “resources” within the capability approach [52]. Resources can be interpreted as individual agency-based capability inputs which, together with structure-based capability inputs, can be converted into capabilities [53]. In the Choice Framework age, gender, ethnicity etc. are conceptualised as personal characteristics of an individual which may in a given social context become related to socially constructed axes of exclusion and influence the scope and scale of the resource portfolio. The resource portfolio consists of: Material resources: These sum up the material objects owned, such as machinery, computer hardware and other equipment. They are also essential inputs in the production process. Financial resources: These stand for financial capital in all its forms (cash, savings, shares etc.). The ability to obtain credit is a combination of the structural character of the banking rules and individual collateral. Natural resources: This includes issues such as geomorphological and climatic conditions in a locality and related aspects such as soil quality and the availability of or access to water as well as the attractiveness of the surrounding nature. Geographical resources: Covers the practical implications of location and relative distances, and also includes the intangible qualities of a location alluded to by writers from Marshall (who refers to the mysteries of the trade “in the air” [54]) to Storper and Venables (who describe the “buzz” of

For examples, see the collection by Fitzgerald and Rowley [57]


Thus, these ten types of resources – material, financial, natural, geographical, psychological, cultural, social, and educational (education and skills) resources; health; and information – represent an attempt to holistically map aspects of the agency element of the systemic framework. However, it is important to recognise that this resourcebased agency can only be realised within the confines of and in systemic interaction with a given structure. This aspect of the Choice Framework will be analysed in the following section. D. Structure Both the empowerment framework suggested by Alsop and Heinsohn and the SLF take into account not only individual agency, but also structures which aid or constrain this agency. Alsop and Heinsohn list “formal and informal laws, regulations, norms and customs” [62] as elements of this structure, while the DFID SLF includes these as laws and “culture” – the latter running the risk of being used as a kind of black box into which all locally specific aspects can be subsumed. The SLF includes not only laws, but also policies, institutions and processes. Rules, laws, norms and policies are embedded in, and often emanate from discourses, and hegemonic discourses can define the thinkspace in which policies, including ICT policies, can be conceived. Thus discourses are included as part of the structure element of the Choice Framework. In particular with respect to ICTs, relevant elements of the structure which influence an individual’s agency include dimensions of access, such as availability, affordability and capabilities needed for using different ICTs [63]. To avoid confusion with Sen’s use of the word, the term “skills” is used instead of capabilities. These dimensions of access are nationally and often locally specific, path dependent and embedded with other elements of the structure. Structural factors such as these stand in a complex relationship with an individual’s resource portfolio. For example, with the help of social resources an individual might have access to the internet (at a neighbour’s house) which might lead to frequent email contact with a distant family member, thus increasing occasions of, in Bourdieu’s terms, legitimate exchange with both the neighbour and the distant relative, in turn potentially increasing social resources. Similarly, a person with higher educational resources (skills and education) and information might find it easier to use the existing access facilities to enhance their skills and gain information. The interface between the opportunity structure and individual agency thus includes a host of reciprocal and cumulative processes. Structural constraints need to be recognized as being as important an element as individual agency. To reflect this, structure is placed above agency in the

diagram of the Choice Framework. The Choice Framework is an attempt to operationalise the capability approach in a holistic and systemic way, thus maintaining much of its conceptual richness. While it may prove particularly useful in the area of ICT4D, the framework could also be applied in other areas of development work.

IV. IMPLICATIONS FOR METHODOLOGY The basic challenge that the capability approach offers to the orthodox methodologies of development research, and ICT4D in particular is that, on a fundamental level, it questions the validity of outcomes that are defined a priori and without consulting the individual in question. Both the inclusion of a development goal and its position within a set of development priorities, however, relate to the question which kind of life people would choose to live and this, according to Sen, is what development is about. A funding institution or government may set, say, economic prosperity as the top priority. Once basic needs such as food and shelter have been met, however, an individual may value being close to family members above earning more money, or may value a healthy environment for themselves and their children over economic growth. In the practice of development projects, this means that before undertaking an intervention designed to improve people’s lives and later measuring its effectiveness, practitioners and researchers would have to ask individuals about their own development priorities and let these guide the planning, implementation, monitoring and evaluation of development projects and programmes. Work done in this area includes participatory monitoring and evaluation [64] and, in a broader context, initiatives around participatory budget planning [65]. Setting development priorities in a participatory way may make the process of development planning more complicated, but it comes with major benefits: Firstly, morally it is the right thing to do to engage the people themselves in the decisions that will affect their lives. Secondly, if the outcomes have been agreed upon in a participatory way, they are more likely to be locally and culturally appropriate and may reduce the rate of failure. Thirdly, a participatory process will lead to greater local buy-in to measures and therefore higher future institutional sustainability. Fourthly, such a process harbours the chance that the current overly economistic focus of development work can be broadened to include environmental, social and cultural aspects and thus better mirror the diversity of the kinds of things people value in their lives. Last, and perhaps not least, ICT and development practitioners work with multi-purpose technologies which offer far more significant changes to people’s lives than the economic impact they have been proven to have. Moving away from an a priori, top-down and often overly economistic


set of development priorities offers the chance to recognise the diversity of the contributions ICTs can make to the social, cultural, environmental and economic aspirations individuals may have for their lives. The Choice Framework is one way of conceptualising such choices by the individual. There are some specific implications for research design which result from the model: A focus on the individual’s own development outcomes means that the research needs to start from these wished-for outcomes, measure the degree to which they have been attained and work systemically backwards through structure, agency and choice to understand how these outcome have come about. The extensive list of resources covers six less tangible resources (social, cultural, educational, psychological resources, health and information) which pose challenges to measurement but need to be taken into account. On the structure side, access to ICTs is conceptualised as availability, affordability and necessary skills. ICTs are embedded in the wider set-up of institutions, policies, programmes, norms and discourses. As such they need to be analysed as firmly and historically engrained in the societies they affect. V. APPLYING THE FRAMEWORK: IMPACTS OF TELECENTRES IN RURAL CHILE The following example is part of an extensive ethnographic study of how state ICT policies affected microentrepreneurs in rural Chile [66]. When interviewing microentrepreneurs and their partners who were using a telecentre located in a public library in rural Chile, open-ended questions revealed that apart from business-related usages such as looking up prices of machinery on the internet (carpenters), looking up photos of furniture models (carpenters), communicating via email with a supplier (carpenters, spice vendor) or buyer (spice vendor), looking up recipes (cake vendor), and looking up guidelines for government business assistance (carpenters), there were several answers which reflected what in many studies of telecentre usage is described as “personal usage”. Respondents’ faces lit up when they described how they now could exchange emails or chat with relatives who were living abroad, children who were studying in a larger town or family members who were working as temporary labour on fruit farms or in the mines in the north of Chile. One respondent, a woman in her 50s whose household income was around 440 USD per month and who together with her husband ran a carpentry business, described how while the Word Cup was on in Germany in 2006 she visited the world cup site to find links and take virtual tours of some of the German cities she was not able to visit in person. As a young woman, she had had a pen friend from Kaiserslautern, and while he had come to visit her in Chile, her dream of visiting him had never been possible because, she said, the money she saved had been spent on her children’s education. Eventually, they had lost touch, but now, she told me with tears in her eyes, over 25 years later, she was finally able to “visit” Kaiserslautern, right

here in the telecentre. If one were to apply a typical questionnaire on telecentre usage to this case, this woman’s usage experience might be subsumed in the category “personal usage” or “other”. Yet the following section will offer a careful application of the Choice Framework to this case: A. Outcome The primary outcome was that the respondent had improved choice, in this case, between “no visit to Kaiserslautern” and “virtual visit to Kaiserslautern”. The secondary outcome achieved was defined by the individual: “to see more of the world” – which in this case, translated into “virtual visit to Kaiserslautern” - in Sen’s terms an “achieved functioning”. The aspiration was “visit to Kaiserslautern”, which since it is feasible, could be seen as a “capability” in Sen’s terms. The achieved functioning ”virtual visit” is not equal to the capability “visit in person” but it is an improvement in outcome over no visit at all. Studies of development outcomes, or more commonly of impacts, often operate with a set of impacts as defined by the funding body, government, international organisation or commercial sponsor. This set of impacts then acts as a checklist informing the construction of questionnaires and interview guides, possibly with some scope for “other activities”. Sen’s approach, with the individual’s choice as the primary outcome, however, would suggest that the analysis needs to start from the ground up, asking people about what lives they value and what outcomes they want to see. For this individual, one of the greatest impacts the telecentre had made was that it had given her the chance to virtually visit Kaiserslautern, something few policymakers or researchers would have predicted. Indeed, some might question whether this is a valid “development outcome” or “impact” for a telecentre. In Sen’s approach, expressed via the Choice Framework, it is. B. Agency The individual in question was a Chilean-mestizo 5 woman in her 50s, married with four children who were all grown up now and had left the home. Her material resources did not include a computer and internet access at home and her financial resources made it difficult for her to spend money on using a computer in the local cybercafés. However her social resources (contacts with friends) had helped her gain the information that there was free access to the internet available at the telecentre in the local library. Her geographical resources (the location of her house) and her state of health were such that she could easily reach the telecentre on foot. She had the cultural resources to not feel intimidated when
5 i.e. not considered part of the indigenous minority which had historically been discriminated against.


entering a space like a library and to know the behavioural code there. With the help of her social resources (knowing the librarian who was now also the director of the telecentre), her educational resources (literacy, rudimentary English) and her psychological resources (extrovert, willingness to ask questions) she quickly learnt how to use the computers. The information she gained online, together with her psychological resources (curiosity, tenacity) allowed her to understand the choices she had and find the site which offered the virtual tour of Kaiserslautern, thus achieving her chosen development outcome. C. Structure The agency of the individual is a shaper of, and is shaped by the structure in which it operates. In this case, as part of the national ICT policy, the Agenda Digital, the state of Chile had signed an agreement with the Bill and Melinda Gates Foundation for them to provide 9.2 million USD worth of hardware to be installed in public telecentres based in libraries around the country, running Microsoft software. The local library was an existing institution which was able to accommodate the hardware, delivered as part of the Biblioredes telecentre programme (availability of ICTs). The Chilean digital literacy campaign provided free IT courses to adults, and public discourse in Chile stressed the importance to become “digitally literate”, so the woman had taken the course (necessary skills). The formal rules for users of the telecentre stipulated that access was free (affordability of ICTs) but limited to 30 minutes per person at busy times, and the informal rules were that people were left to use the computers on their own unless they asked for help from the telecentre director. Norms on the usage of space made it easy for her to go to a library as a mestizo woman with a completed school education who was known in town. However, norms on the usage of time meant that she could only use the telecentre when she was not supposed to be home preparing meals (gendered norms on time) or when she was expected to attend to customers (business norms on time). D. Dimensions of choice In the Choice Framework, an individual’s resource-based agency can operate within a given structure to achieve degrees of empowerment, such as existence of choice, sense of choice, use of choice and achievement of choice. In this case, both the choice “travel to Kaiserslautern in person” and “take a virtual tour of Kaiserslautern” existed, the latter only since the links were offered via the World Cup website in 2006. In a capitalist market system, however, the former choice required an amount of financial resources which the individual felt unable to dedicate to this idea. The choice “take a virtual tour of Kaiserslautern” however, required a good internet connection, a computer, the knowledge that the tour was available via the website, the skills to find and run it, and time. The individual, thanks to among others, her social resources,

information, and psychological resources, knew that the telecentre offered a computer and a good connection, had acquired the skills to navigate the Internet and run an application in the free digital literacy courses offered at the telecentre. She felt that informal, gendered social norms allowed her to go to the telecentre during the morning before having to prepare lunch. Thus she developed a sense of choice, was able to choose (use of choice) and achieved her desired outcome (achievement of choice). VI. CONCLUSION Applying the Choice Framework to this particular case allows us to firstly, theorise the use of ICT in a systemic and procedural way which reflects the systemic and pervasive impact of ICT. The “impact of ICT” is not conceptualised in a cause- and effect chain, instead effects are carefully disaggregated and their systemic interrelatedness and cocausality is demonstrated. Secondly, the Choice Framework offers a way to operationalise Sen’s capability approach in the context of ICT. Sen’s approach is currently the most well known heterodox alternative to orthodox, growth-focused and often economistic conceptualisations of development. Given the enormous potential of ICTs to give individuals choices, and indeed a greater sense of choice, Sen’s approach is of particular interest to the ICT and development research community. There are three obvious limitations to the application of the Choice Framework, and this is where more theoretical work needs to be done: Firstly, the Choice Framework aims to be comprehensive in its modelling of the complex relationships between agency, structure, degree of empowerment and outcome, and this automatically entails a trade-off with the depth of theorisation of each element. Behind each of the terms included in the framework lies a wealth of theoretical literatures which may need to be synthesised for different research purposes and key issues brought to the attention of researchers in the development field. While for example, social resources can be theorised by linking to the wider debate on social capital (see [66]), which has been received in the development studies discourses, work on cultural capital (in Bourdieu’s sense) is hardly ever linked to development discourses in the South. Secondly, the Choice Framework is relatively easily applied in qualitative work on the micro-level of the individual. A further challenge will be how to apply the framework to groups of individuals, communities, or even nations. Within this and related to a theoretical tension evident in Sen’s original approach, there is a complex relationship between individual and collective choice which will have to be conceptualised carefully.


The example used here to illustrate the potential of the Choice Framework is part of a far more extensive ethnographic and longitudinal study of how the state ICT policies affected microentrepreneurs in rural Chile [66]. In this study, connections are made between individual and collective choice. For example, the local authority’s public procurement policy was analysed as an expression of collective choice, but related to individual’s views as to how their tax money should be used to create the community they wanted to live in. However, further empirical work is necessary to gather experiences in the applicability of the Choice Framework in other cultural and socio-economic settings. The relationship between individual choice and collective choice needs to be conceptualised carefully in these local contexts in order to allow for the empirical application of the Choice Framework at the more aggregate level such as socalled target groups and communities. The third limitation is a very practical one: funders prefer predefined and clearly measurable impacts. The Choice Framework, however, suggests that impacts of ICTs occur in a systemic, pervasive and transversal way, and that outcomes should be defined, in line with Sen’s approach, by the individual, based on their choices as to what kind of life they value. There are, however, some funders who are open to methods such as participatory evaluation and monitoring, and this means there might be hope for genuinely people-centred development work – and development theory. There are some key implications emerging from this theoretical work for practitioners of ICT. Firstly, while no technology is ever completely politically neutral [67][68], ICT4D projects can be placed on a continuum of “directional control”. At one end there are projects and programmes which focus on providing people with access to a technology which is recognised as multi-purpose, like some telecentre projects. On the other end of the continuum are projects and programmes which carry a much more narrow set of intentions, for example training microentrepreneurs to use a specific e-procurement system in order to “train them” to operate in a more competitive market environment under a specific set of rules [67]. The further down the directional control continuum a particular project and programme is located, the more risk there is that the intended outcomes of an ICT4D project diverge from the capabilities, or desired outcomes individuals in the so-called target group would choose. Thus, the more directional control is involved in the project or programme, the more participation of the set of individuals who are the intended group will be needed to reduce this gap. This would include conceptualising the development process as open-ended and the so-called target group as individuals empowered to choose the lives they themselves value. Participatory project design and participatory monitoring and evaluation techniques would be most appropriate.

Secondly, there are some macro-methodologies which reflect the ethos of giving people the power to choose. Voucher schemes are a good pragmatic way to monitor, in a heavily supply-driven development field such as ICT4D, what products (hardware, software etc), services (trainings, computer repair, communication etc) and content (economic, social, political, cultural etc) people would, after considering their options, actually choose. From the field of participatory urban planning come methodologies for participatory budget design, where communities get to debate and decide which of their desired outcomes to prioritise and pursue. This is a practical and democratic way to aggregate individual capabilities in order to enable collective decision making, and could also be used for ICT4D. Thirdly, practitioners may deduce that if the ideal is for development projects’ intended outcomes to reflect the individual’s choices, then the more individuals are aggregated to a group, the less probable it is that they can agree on a similar set of capabilities. From this follows that the further down the directional control continuum an ICT4D project is, the more sensitive/locally customised it has to be to the choices of a smaller number of people. Big, uni-directional development programmes with specific, a priori defined desired outcomes designed for a large number of people are most likely to be in contradiction to a people-centred holistic development process as proposed by Sen and expressed in the Choice Framework. Ultimately, researchers working on ICT, particularly the internet, need to consider the question: Should we try and fit a groundbreaking, multi-purpose and potentially liberating technology into orthodox notions of development – such as more ICT for higher GDP, more ICT for better school results et cetera – impacts which we may struggle to prove? Or can the field of ICT and Development serve as a test case and breeding ground for thinking about development in a more holistic way, putting the individual and their own choices at the centre of development? If the latter is the case, then we have plenty of work to do, but the Choice Framework may serve as one part of the big puzzle we have to begin putting together. ACKNOWLEDGMENT I would like to thank Diane Perrons, Robin Mansell, Alexandra Norrish, Macarena Vivent, Rodrigo Garrido, Tim Unwin, and four anonymous reviewers for their comments on earlier versions of this paper.

Chile, unpublished PhD thesis, Department of Geography and Environment, London, London School of Economics, 2007 C. Moser, Gender Planning and Development, London, Routledge, 1991 W. Van Eyken, The Concept and Process of Empowerment. The Hague, Bernard van Leer Foundation, 1991 J. Friedmann, Empowerment: The Politics of Alternative Development, Oxford, Blackwell, 1992 N. Kabeer, “Resources, agency, achievements: reflections on the measurement of women’s empowerment”. Development and Change, vol. 30, no.3, pp.435-464, 1999 P. Oakley (ed.), Evaluating Empowerment, Reviewing the Concept and the Practice, Oxford, INTRAC, 2001 A. J. Bebbington, M. Woolcock, M. Guggenheim and E.A. Olson (eds), The Search for Empowerment. Social Capital as Idea and Practice at the World Bank, Bloomfield, CT, Kumarian Press, 2006 R. Alsop and N. Heinsohn, Measuring Empowerment in Practice – Structuring Analysis and Framing Indicators, Washington D.C.: World Bank, p.5, 2005 Ibid, p.6 Ibid, p.8 Ibid, p.9 R. Chambers and G. R. Conway, “Sustainable Rural Livelihoods: Practical Concepts for the 21st Century”, Discussion Paper 296, Institute of Development Studies, University of Sussex, UK, 1992 A. J. Bebbington, “Capitals and Capabilities: A Framework for Analyzing Peasant Viability, Rural Livelihoods and Poverty”, World Development, vol. 27, no.12, pp. 2021-2044, 1999 D. Carney, Sustainable Livelihood Approaches Compared, London, Department for International Development DFID, Sustainable Livelihoods Guidance Sheets, Department for International Development, London, 1999 R. Duncombe, “Using the Livelihoods Framework to Analyze ICT Applications for Poverty Reduction through Microenterprise”, Information Technologies and International Development, vol. 3, no. 3, pp.81-100, 2006 DFID, Sustainable Livelihoods Guidance Sheets, Department for International Development, London, 1999 A. Munasib, Social Capital at the Individual Level: A Reduced Form Analysis. Available: http://munasib.myweb.uga.edu/papers/SocialCapitalDemography.pdf, 2004 D. Kleine, Empowerment and the Limits of Choice: Microentrepreneurs, Information and Communication Technologies and State Policies in Chile, unpublished PhD thesis, Department of Geography and Environment, London, London School of Economics, 2007 A. Sen, Development as Freedom, Oxford, Oxford University Press, 1999 A. Sen, Resources, Values and Development, Cambridge, MA, Harvard University Press, 1984 I. Robeyns, “The Capabilities Approach: An Interdisciplinary Introduction”, Department of Political Science and Amsterdam School of Social Sciences Research Working Paper, University of Amsterdam, Amsterdam, 2003 A. Marshall, Principles of Economics, London, Macmillan, 1920, reprint edition 1961 M. Storper and A.J. Venables, “Buzz: face-to-face contact and the urban economy”, Journal of Economic Geography, vol. 4, no. 4, pp. 351-370, 2004 R. Alsop and N. Heinsohn, Measuring Empowerment in Practice – Structuring Analysis and Framing Indicators, Washington D.C., World Bank, 2005 R. Fitzgerald and C. Rowley (eds.), Human Resources and the Firm in International Perspective, Cheltenham, Edward Elgar, 1997 R. Heeks, “Information and communication technologies, poverty and development”. Development Informatics Working Paper No.5, Manchester, IDPM, 1999 B.S. Gigler, “Including the excluded – can ICTs empower poor communities? Towards an alternative evaluation framework based on the capability approach,” presented at the Fourth International Conference on the Capability Approach, Pavia, Italy, Sept. 5-7, 2004 P. Bourdieu, “The forms of capital”, in: J. Richardson (ed.), Handbook of Theory and Research for the Sociology of Education, New York, Greenwood Press, 1986

[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] R. Heeks, “Information systems and developing countries: Failure, success and local improvisations” in The Information Society, vol. 18, no. 2, pp. 101-112, 2002 W.A. Lewis, “Economic development with unlimited supplies of labour”, The Manchester School of Economic and Social Studies, vol.22, no. 2, pp.139-91, 1954 G. Myrdal, Economic Theory and Underdeveloped Regions, London, Duckworth, 1957 A. O. Hirschman, The Strategy of Economic Development, Yale, Yale University Press, 1958 W. Rostow, The Stages of Economic Growth: A Non-Communist Manifesto, Cambridge, Cambridge University Press, 1960 A.G. Frank, Capitalism and Underdevelopment in Latin America, London, New York, Monthly Review Press, 1967 T. Dos Santos, Dependencia y Cambio Social, Santiago, Universidad de Chile, 1970 M. Blomström and B. Hettne, Development Theory in Transition. The Dependency Debate and Beyond: Third World Responses, London, Zed Books, 1984 C. Kay, Latin American Theories of Development and Underdevelopment, London, New York, Routledge, 1989 M. Nerfin (ed.), Another development: Approaches and Strategies, Uppsala, Dag Hammarsköld Foundation, 1977 R. Chambers, Rural Development: Putting the Last First, Harlow, Longman, 1983 A. Escobar, Encountering Development: The Making and Unmaking of the Third World, London, Longman, 1995 J. Nederveen Pieterse, “After post-development”, Third World Quarterly, vol. 21, no.2, pp.175-191, 2000 A. Sen, Development as Freedom, Oxford, Oxford University Press, 1999, p.3 J. Harriss, Depoliticizing Development – The World Bank and Social Capital, London, Anthem Press, 2001 A. Sen, “Equality of what?”, in S. McMurrin (ed.) The Tanner Lectures on Human Values, Salt Lake City, 1980 A. Sen, Resources, Values and Development, Cambridge, MA, Harvard University Press, 1984 A. Sen, “Capability and well-being”, in M. Nussbaum and A. Sen (eds.) The Quality of Life, Oxford, Clarendon Press, 1993 A. Sen, Development as Freedom, Oxford, Oxford University Press, 1999 A. Sen, Development as Freedom, Oxford, Oxford University Press, 1999, p.3 M. Nussbaum, Women and Human Development: The Capabilities Approach, Cambridge, Cambridge University Press, 2000 S. Alkire, Valuing Freedoms, Oxford, Oxford University Press, 2002 D. Clark, Visions of Development, Cheltenham, Edward Elgar, 2002 I. Robeyns, “Sen’s capabilities approach and gender inequality: selecting relevant capabilities, Feminist Economics, vol. 9, no 2-3, pp. 61-92, 2003 I. Robeyns, “The Capabilities Approach: An Interdisciplinary Introduction”, Department of Political Science and Amsterdam School of Social Sciences Research Working Paper, University of Amsterdam, Amsterdam, 2003 R. Sugden, “Welfare, resources and capabilities: a review of Inequality Re-examined by Amartya Sen”, Journal of Economic Literature, vol. XXXVI, pp. 1947-1962, 1993 J. Roemer, Theories of Distributive Justice, Cambridge, Cambridge University Press, 1996 A. Sen, “Capability and well-being”, in M. Nussbaum and A. Sen (eds.) The Quality of Life, Oxford, Clarendon Press, 1993 A. Sen, “Maximisation and the act of choice”, Econometrica, vol. 65, no.4, pp.745-779, 1997 A. Sen, Inequality Re-Examined, Oxford, Clarendon Press, 1992 R. Alsop and N. Heinsohn, Measuring Empowerment in Practice – Structuring Analysis and Framing Indicators, Washington D.C.: World Bank, 2005 D. Kleine, Empowerment and the Limits of Choice: Microentrepreneurs, Information and Communication Technologies and State Policies in

[33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47]

[48] [49]


[51] [52] [53]

[54] [55] [56] [57] [58] [59]

[26] [27] [28] [29] [30] [31] [32]


[61] Ibid, p.249 [62] R. Alsop and N. Heinsohn, Measuring Empowerment in Practice – Structuring Analysis and Framing Indicators, Washington D.C., World Bank, 2005, p.9 [63] R. Gerster and S. Zimmermann, Information and Communication Technologies (ICTs) for Poverty Reduction?, Bern, Swiss Agency for Development Cooperation, 2003 [64] I. Guijt and J. Gavena, Participatory Monitoring and Evaluation: Learning from Change, Brighton, Institute of Development Studies, 1998 [65] A. Shah (ed.) Participatory Budgeting. Washington D.C., The World Bank, 2007. Available: http://siteresources.worldbank.org/PSGLP/Resources/ParticipatoryBudg eting.pdf [66] D. Kleine, Empowerment and the Limits of Choice: Microentrepreneurs, Information and Communication Technologies and State Policies in Chile, unpublished PhD thesis, Department of Geography and Environment, London, London School of Economics, 2007 [67] D. Kleine, “The ideology behind the technology – Chilean microentrepreneurs and public ICT policies”, Geoforum, vol. 40, no.2., 2009 [68] D. Kleine, “Striking a Balance”, Engineering and Technology, February 2007, p. 30-33


ICTD for Healthcare in Ghana: Two Parallel Case Studies
Rowena Luk, Matei Zaharia, Melissa Ho, Brian Levine, and Paul M. Aoki

Abstract — This paper examines two parallel case studies to promote remote medical consultation in Ghana. These projects, initiated independently by different researchers in different organizations, both deployed ICT solutions in the same medical community in the same year. The Ghana Consultation Network currently has over 125 users running a Web-based application over a delay-tolerant network of servers. OneTouch MedicareLine is currently providing 1700 doctors in Ghana with free mobile phone calls and text messages to other members of the medical community. We present the consequences of (1) the institutional context and identity of the investigators, as well as specific decisions made with respect to (2) partnerships formed, (3) perceptions of technological infrastructure, and (4) high-level design decisions. In concluding, we discuss lessons learned and high-level implications for future ICTD research agendas. Index Terms — Remote medical consultation.

I. INTRODUCTION ICTD research focuses primarily on matters relating to its target populations and their conditions, such as how individuals, communities and institutions interact with technology. For the most part, investigators only become part of the scene as objects of retrospective critique, usually in the context of having failed to see some vital point and achieve some critical goal. As with any kind of research, however, ICTD research is a product of social, cultural and professional influences on the investigators, influences that affect every decision made over a project’s lifetime. It is extremely difficult to reflect upon the effects of these influences because each project is so uniquely framed by the contexts of its investigators and its target setting that it is hard for analysts to imagine how “it could have been otherwise” [19]. In this paper, we address this reflective gap through a sideManuscript received September 22, 2008. This material is based in part upon work supported by the U.S. National Science Foundation under Grant No. 0326582. M. Zaharia is supported by the Natural Sciences and Engineering Research Council of Canada R. Luk is with AMITA Telemedicine Inc., Toronto, ON, Canada (e-mail: rowena@amitatelemedicine.org). M. Zaharia is with the Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA (e-mail: matei@berkeley.edu). M. Ho is with the School of Information, University of California, Berkeley, CA, USA (e-mail: mho@ischool.berkeley.edu). B. Levine was with New York University, New York, NY. He is now with the Department of Obstetrics and Gynecology, New York Presbyterian Hospital, Columbia University Medical Center, New York, NY, USA. (email: DrBrianLevine@gmail.com). P.M. Aoki is with Intel Research, Berkeley, CA, USA (e-mail: aoki@acm.org).

by-side analysis of two completely independent projects which arose from similar objectives but resulted in two very different strategies. Two sets of researchers came to the same country to work with the same community, identified the same problems, and proposed two different solutions. Both projects were implemented over the course of 2007-2008, both aimed to tackle remote medical consultation among Ghanaian doctors, and both sought nationwide deployment over the course of the project lifetime. However, the Ghana Consultation Network (GCN) was initiated by a group of technologically-oriented researchers, while the OneTouch MedicareLine (ML) was initiated by a public health researcher and social entrepreneur. GCN’s solution consisted of a Web application hosted on a delay-tolerant network of computers running in each hospital and on the open internet. ML focused instead on a mobile phone program involving a combination of technological services and business innovation. We discuss the rationale behind decisions made by each party over the course of the project and the resulting outcomes. The main contribution of this paper is to reflect on some ways in which the context and background of researchers affect the structure of ICTD projects. A second contribution is in identifying relevant factors which will assist in the design and execution of ICTD projects in the future. In particular, we provide specific examples of how an investigator’s institutional context and identity affect not only the methodology used but also the interpretation of findings; how the partnerships chosen for co-development and codeployment have a fundamental role in the development and deployment of the technology; how ‘objective’ assessments of existing technology infrastructure are influenced by personal areas of expertise; and how projects can be designed with various degrees of technological ‘specificity’ and the resulting implications for their impact on various development indicators. We expect these findings to be of interest to the community of ICTD researchers and practitioners as a whole. The paper is organized as follows. We first present a background overview of some of the underlying issues of healthcare in Ghana and briefly introduce both projects. We then discuss in turn the framing of the research problem and how it affected the partnerships formed and resulting deployment strategies; the assessments of technological infrastructure by the project members; and the usage and appropriation of the two solutions. Finally, we discuss highlevel implications for ICTD research and present works related to each of these sections.


II. BACKGROUND In this section, we highlight some of the background issues of healthcare in Ghana and provide an overview of the two projects which are the focus of our case studies. A. Healthcare in Ghana Like many other countries, Ghana has a tiered healthcare system in which cases that cannot be handled by an institution at a given tier are referred to institutions above it in the hierarchy. All referral chains culminate in one of two teaching hospitals, both located in the South, where specialists have the training and resources necessary to carry out more complex procedures. Doctors, and in particular specialists in areas such as internal medicine, are highly concentrated in the urban South. This sometimes leaves only two or three doctors to serve in district hospitals in the rural North; predictably, rural doctors are confronted with heavy workloads (meaning they can spend only a few minutes face-to-face with a given patient per day) and isolated working environments (which prevent them from taking advantage of many of the educational and collaborative programs available to doctors in the South). Formal and informal consultation is highly integrated into the life of all doctors. Between hospitals, many doctors call personal contacts – friends, colleagues, classmates – to seek advice, and within hospitals this behavior is even more frequent. Such “curb-side” consultation has long been a common observed characteristic of medical practice [16] but it is of critical importance in environments where specialist expertise is spread thinly. Continuity of care is difficult to ensure. Patient records are almost universally paper-based. Despite attempts by some major hospitals to transition towards an electronic system, none of the 18 hospitals visited had a working system. Within a major hospital, there is nothing to guarantee complete patient records besides a doctor’s own discretion and multiple reminders from administration. It is not uncommon for records to be lost, confused, or incomplete. The communication infrastructure in the country is severely limited. Hospital landlines are frequently out of service, forcing many doctors to rely on personally-purchased mobile phones. Likewise, broadband Internet infrastructure is unevenly available, frequently unreliable where available, and dependent on hospital budget allocations. Other options are available throughout the country, such as dial-up, satellite, and mobile data plans, but these are generally expensive, unreliable, and/or slow as well. B. The Ghana Consultation Network (GCN) Solution GCN is an end-to-end, computer-based system providing doctor-to-doctor medical consultation on a network of servers implementing a distributed, asynchronously-synchronized database (Figure 1) [20]. The goal is to allow doctors throughout Ghana to consult with each other as well as medical professionals (in particular, the large Ghanaian medical diaspora) around the world. Doctors access the system through a Web-based UI (Figure 2), either by logging

Figure 1. GCN distributed, delay-tolerant server network
(c) new thread (c) new thread (a) primary case list (a) primary case list

(d) navigation menu (d) navigation menu

(b) “other cases” list (b) “other cases” list cases”

Figure 2. Key elements of GCN’s welcome screen

into a local server (hosted at some participating hospitals) or by logging into one of the two public servers (hosted with Internet service providers (ISPs) in both Ghana and the U.S.). Providing local servers ensures availability and responsiveness to the users in the face of unreliable network connectivity and makes the task of synchronizing data between servers transparent to them. Synchronization is automated and carried out over a disconnection-tolerant messaging layer called OCMP [29] which draws inspiration from the research into delay-tolerant networking (DTN) [7],[12]. Because the doctors already view consultation as a matter of reaching out to personal contacts, the system is presented as a social networking platform – a forum for medical consultation with social and professional colleagues – and leverages social incentives using principles drawn from the HCI and CSCW communities [20]. The system supports two types of ‘conversations’: highly structured ‘consultations’ for specific patients (which work much like an electronic case history) and unstructured ‘discussions’ (which work much like online


forums). So, for example, a doctor unsure of how to treat a particularly resistant case of tuberculosis might fill out various fields of the ‘consultation,’ address it to a colleague, and wait for advice to be appended to that consultation, whereas another interested in general updates on malaria treatment might create a question under ‘discussions,’ addressed to any interested doctor. Recognizing the ubiquity of mobile phones, GCN also incorporates text message (as well as email) notifications of new content; yet the core of the interaction is designed for computers. This is based on feedback during the design process that the wealth of data required for patient management would demand a screen size larger than those present in mobile phones. To date, over 125 doctors have been enrolled from Ghana, the U.S., Mali, Nigeria, South Africa, and the U.K. and 39 consultations have been submitted. Methodology GCN is the product of a conventional user-centered design process. There have been four rounds of iterative design and fieldwork, starting with exploratory needs assessment (2005) and continuing in conjunction with design exercises (20062007), a pilot deployment (2007), and an ongoing deployment (2008). Overall, interviews or focus groups were conducted with 132 doctors in 15 hospitals throughout Ghana [20]. The most recent fieldwork in Ghana lasted six weeks in mid-2008. Exploratory interviews on mobile phones and computer usage were conducted with 35 internal medicine (IM) doctors at a major teaching hospital. Further, six evaluation interviews at one regional hospital and two interviews at one district hospital were conducted (both sites of the original pilot). Limitations of the methodology include an overrepresentation of internal medicine doctors from urban hospitals. While the initial rounds of fieldwork focused on rural hospitals which are more numerous in the North, the current live deployment is centered in the more accessible South, with plans to extend later. We have established public servers in both Ghana and the US, as well as local servers in 3 major hospitals and 2 rural district hospitals in the South. These district hospitals suffer from some of the same problems as in the North – a shortage of doctors and poor travel and communication infrastructure – although the severity of these challenges is much less. Our current strategy is first to recruit specialists from the major hospitals and to test our system in the district hospitals of the South before reaching out to more challenged environments in the North.

Figure 3. ML Phase 1 calls for free phone calls and text messages; the planned ML Phase 2 calls for MMS and data reports over SMS, and ML Phase 3 calls for free smartphones, reference tools, and custom applications.

C. The OneTouch MedicareLine (ML) Solution MedicareLine is a program currently offering free calls and text messages between any registered physician and/or surgeon within Ghana. Its current focus has been on reducing logistical and economic barriers to mobile phone use rather than on technological innovation. After submitting the required paperwork, a doctor registered with the Ghana Medical Association (GMA) receives a OneTouch GSM SIM which can be used with a privately-purchased mobile phone. Using this SIM, the physician can now call other program participants free of personal charge. For example, a physician can call a specialist in the capital or a friend in a rural town to ask or provide medical consultation. This can be a significant cost saving, especially given that airtime in Ghana is relatively expensive compared to many developed countries. The GMA also has a computer terminal that can send “blast” texts to all participants for updates and notifications. Future phases of the program envision new technological interventions (Figure 3). Phase 2, as yet uncompleted, calls for physicians to receive free MMS service so that they can augment their phone consultations with photos (e.g., of a skin condition or X-ray). Phase 2 will also allow the GMA and other government organizations to collect data from physicians via SMS. In Phase 3, ML anticipates partnering with hardware vendors to provide each physician with a smartphone preloaded with medical reference software. Phase 1 of this program has already experienced a very high rate of adoption. Approximately 1700 of 2000 doctors in the GMA have enrolled, with over 2 million calls made to date. Methodology ML arises from a five-week visit to Ghana in October 2007 and a two-week visit in March 2008. The purpose of the first
OneTouch MedicareLine (ML) Mobile phones >1700 doctors in Ghana Cost of cellular airtime Doctors in Ghana GMA and OneTouch (mobile operator) Adequate mobile phone quality and coverage Low – only generic mobile phone service National (available country-wide at launch)

Table 1 – Comparison of GCN and ML Ghana Consultation Network (GCN) Computers Primary platform 89 doctors in 5 hospitals in Ghana Adoption >125 doctors total from around the world Network connectivity Problem addressed Doctors in Ghana and Ghanaian medical diaspora Target user Ministry of Health, GPSF, and KNet (ISP) Partners (see Table 2) Adequate internet connection quality and coverage Assumptions punctuated by regular power and network outages High – custom system software and Web application Specificity (see Section VI) Incremental (hospital by hospital) Deployment strategy


Figure 4. Timeline of GCN and ML Projects, including fieldwork.

visit was to introduce and promote usage of a social networking Web system for international medical collaboration that had already been developed in the States; however, within two weeks of arriving in Ghana, the investigator decided that access to both computers and the Internet was a fundamental problem. Discarding the Webbased project entirely, he moved on to exploratory interviews with all stakeholders, including over 30 doctors, politicians, local businessmen, and administrators, focusing on the issue of barriers to communication, innovation, and current technology usage. He conducted site visits to five regional hospitals and one polyclinic in the urban South, and visited Tamale, a northern regional capital, for four days. Towards the end of that trip, the investigator orchestrated a “meeting of the minds” between the CEO of OneTouch and the head of the GMA, whereupon an initial understanding was reached. Implementation was left to the two partnering agencies, which announced the program the following month and launched it in January 2008. In March 2008, the investigator returned to conduct another round of unstructured interviews with over 20 individuals, again from diverse communities, in order to assess the program’s progress. In addition to ML’s own follow-up interviews, material for this paper also comes from 10 interviews conducted by the GCN investigators with doctors at one of the teaching hospitals, as well as their interviews with the ML investigator directly. III. THE ROLE OF PROJECT CONTEXT IN FRAMING PROBLEMS ICTD work frequently involves the interdisciplinary participation of various communities of research and of practice [5], but what are the implications of working within these communities, given similar projects with similar goals over similar timeframes? Both projects clearly did some amount of fieldwork in both the urban South and the rural North, both purportedly wanted to develop a solution that would work for all Ghanaian doctors, but GCN focused on connectivity in the rural North using innovative technology while ML focused on building communication in the urban South using proven technology. In this section, we examine the institutional, cultural, and personal contexts from which each project arose.

A. GCN GCN began as part of an ongoing collaboration between a U.S. corporate research laboratory and a U.S. research university. Specifically, it was a project of a joint research group which had worked extensively in the area of low-cost connectivity and delay-tolerant networking. Project funding came from U.S. government grants as well as the corporate sponsor. In the first brainstorming/conception phase of this project, the axes by which ideas were evaluated were defined as: (1) direct social impact (e.g., improvements in healthcare delivery), (2) medium-term impact on ICT adoption in developing regions (e.g., finding novel ways to make ICT more relevant in addressing local problems), and (3) long-term contribution towards HCI research. The primary short-term deliverables were software, real deployments, and research papers. As for the individual investigators, all came from a technology and research background, collectively with experience in systems, networks, and HCI. B. ML ML began as a project funded by a social entrepreneurship program and the international health program of the School of Medicine at a U.S. research university. As part of the international research and education component of a medical program, there were no short-term deliverables. The individual investigator comes from a medical program with prior experience in medical research and a personal interest in consumer technology. His personal goal was to identify a project with sustainable, wide-spread social impact that had the potential to be financially self-sufficient and reproducible in other developing countries. C. Discussion These differing contexts had far-reaching implications for how the problem was framed. First, coming from a more technological background, GCN investigators were in a better position to attempt technological innovations and, in particular given their knowledge of projects such as [9] and [22], on creative ways of addressing poor connectivity. The ML project’s lack of technological expertise limited the possibility of developing new technology. Second, ML, given its ties to a major U.S. hospital, had a natural predisposition to tackle problems internal to a specialist center, whereas GCN had no expertise in hospital administration and hence was more inclined to address the simpler logistics of the rural clinic. Third, ML’s agenda for large-scale social change predisposed it towards impacting the greatest number of doctors, who are of course concentrated in the urban South with better mobile phone coverage. GCN chose to focus design efforts on the problems of the North (such as those of the 14 doctors servicing half a million people in Ghana’s Upper West region), seeing this as the best use of limited resources. These different problem framings have various implications for the solutions developed and implemented, as we discuss in the following sections.

Table 2 – Healthcare institutions / partners Acronym Name and Description Ghana Medical Association GMA A voluntary association of 95% of Ghana’s doctors, representing their interests nationally Ministry of Health MOH The national health administration, responsible (through the Ghana Health Service) for all public hospitals Ghana College of Physicians and Surgeons GCPS Responsible for specialist education and certification Ghana Physicians and Surgeons Foundation GPSF U.S.-based non-profit organization promoting specialist education in Ghana Figure 5. Map of GCN current and anticipated partners.

IV. PARTNERING STRATEGY AND DEPLOYMENT The question of partner selection is a key one for ICTD deployments, as partners are the usual means by which technology makes it from the laboratory to the field. In this section, we examine the impact of local and non-local partners on technology design and deployment. A. GCN GCN chose partners based on the decision to deploy technology in hospitals to connect their doctors with the Ghanaian diaspora (Figure 5). The Ghana MOH was chosen as a local institutional partner because it had the central authority to allow the servers to be installed in public hospitals. With the initial approval of the MOH, GCN was able to work quickly with the public hospitals, conducting 121 interviews with relatively little trouble alongside iterative, incremental deployment of technology. Of course, the drawback of the centralized approach is that it requires convincing a risk-averse bureaucracy in advance, thereby running the risk of ‘over-selling’ the solution. In addition to the added effort, another unexpected difficulty of working through the MOH was overloaded communication channels. Advertising GCN to the doctors was a challenge because promotional material needed to work through the same mechanisms through which the MOH communicated, for example, minor procedural changes and optional seminars from pharmaceutical companies. So there were multiple instances when administration would send out notifications of GCN training sessions but only a small fraction of doctors would show up. GCN partnered with the Ghana Physicians and Surgeons Foundation (GPSF), a Ghanaian medical diaspora organization, to recruit medical consultants in the U.S. KNet, a small Ghanaian Internet service provider (ISP), provided local Web hosting. B. ML ML worked with a smaller number of partners, all nongovernmental (Figure 6). ML chose the GMA as a local institutional partner because it represented the interests of physicians and its leaders would immediately see the benefit of the project to the physicians as individuals, if not to the healthcare system as a whole. The program is framed as a GMA value-added member service (similar to other member benefits such as professional development seminars and quarterly publications). In contrast to the MOH information

Figure 6. Map of ML current and anticipated partners.

channels, doctors took personal initiative to read the GMA emails or check GMA bulletin boards; this lead to widespread awareness of ML. OneTouch was chosen as a technological service provider for political and pragmatic reasons. As the investigator stated: OneTouch was the national company. I wanted this to be Ghanaian – by Ghanaians, for Ghanaians. That says a lot more than ‘done by Ghanaians in concert with South Africans.’ [2] OneTouch had the means and the expertise to very quickly make the ML program a reality and the financial resources to support the program independently on an ongoing basis. Thus, while GCN’s deployment was incremental, ML’s was all-ornothing, a strategy consistent with ML’s social agenda for rapid, wide-scale impact. Of course, the drawback of working entirely through OneTouch was that the project was no longer under the control of the investigator but was now subject to the organizational vagaries of a for-profit corporation. That is, changes in OneTouch’s business priorities could result in the program being dropped as quickly as it was initiated. This question is immediately salient in light of the recent acquisition of OneTouch by a multinational carrier based outside of Ghana (in fact, in South Africa). C. Discussion In considering the non-local partners, GCN and ML bear striking similarities in that both were initiated in partnership with U.S.-based organizations hoping to foster communication with the Ghanaian medical community: GCN with GPSF and ML with a U.S. teaching hospital. However, GCN emphasized the role of GPSF throughout the course of its lifetime whereas ML stopped working with U.S. doctors in order to focus explicitly on the Ghanaian context. The investigator noted:


[B]y removing this component of having this international cross dialogue, and realizing that we needed to have intraGhanaian communication instead of inter-Ghanaian communication, I quickly came to the conclusion that I needed to do something to improve communication within Ghana. [2] The relevance of overseas medical consultants was ultimately one of the core reasons that ML could focus on mobile phones while GCN retained a focus on the Internet. With respect to deployment, this meant that GCN had the additional task of advertising at GPSF conferences in the U.S. and to other organizations in the West, diluting its efforts in Ghana. GCN’s partnership with the MOH, GPSF and KNet and ML’s partnership with the GMA and OneTouch were factors in the projects’ very different rates of adoption. Arriving in the country with equipment for the initial pilot deployment, GCN was deployed in four hospitals over the course of five weeks, garnering an initial user base of 73 doctors. Over a similar stretch of five weeks, MedicareLine went from being a conversation between two CEOs to a national program, and by four months later over 1600 doctors had used ML to make over a million phone calls across the country. On the topic of sustainability, the juxtaposition of these two projects raises some interesting questions surrounding the rhetoric of “organic adoption” and its impact on long-term sustainability of ICTD initiatives. That is, they reflect two different views of what ‘organic’ or ‘bottom-up’ adoption means. ML presented a more ‘organic’ adoption model in the sense that the program was announced and doctors could sign up according to their individual needs and interest. On the other hand, because one institution provides all of the technical and financial resources, there is less local ownership and control over the maintenance of each project. Essentially, there is no guarantee that OneTouch (or the GMA) will not unilaterally end the project. This creates a situation analogous to the many development projects which rely on inconsistent or limitedterm donor funding. GCN’s model involves a more laborious adoption process but benefits from complete in-hospital ownership. This is a different type of ‘organic’. This ‘organic’ involved more than just the end user; it involves the whole system of people and machines that need to be in place for this network to grow. Unfortunately, this also surfaces the issue of the lack of access to ICT expertise experienced by all but the largest urban hospitals; the benefits of decentralized ownership are compromised by the geographic concentration of ICT skills. GCN can be conceived as a centralized project (sponsored by MOH and adopted by hospitals) with decentralized deployment (core resources provided by hospitals) while ML is a decentralized project (adopted independently by individual doctors) with centralized deployment (core resources provided by OneTouch). Technology innovators (such as corporations) often exhibit a bias towards decentralized solutions distributed through markets. Yet there are many examples, of which GCN would be one, of a potentially valuable tool which does not become useful until it has sufficient infrastructure and

“network effects” to fulfill its potential. Balancing the costs and benefits of the approaches remains an open question. V. ASSESSING TECHNOLOGICAL INFRASTRUCTURE One might think that doing a baseline assessment of available technology infrastructure would be one of the most objective elements of an ICTD project, one that is a basic part of requirements analysis. In this section, we observe that GCN and ML drew on two very different sets of infrastructure assumptions and show that such assessments are highly influenced by personal and institutional context. A. GCN The GCN researchers designed the system around the pessimistic engineering assumption that the system must continue to function under the worst-case connectivity conditions in the rural North – i.e., that all options available are unreliable and often low-throughput – and optimistic assumptions about PC usage. The exploratory needs assessment fieldwork found that the quality of mobile phone connectivity was unacceptable in the North. It is generally so oversubscribed that if you are calling a mobile phone from another service provider, you need to dial 10-15 times in order to successfully connect. [1] GCN made no quantitative measurement of the quality and reliability of network connectivity, instead relying on qualitative descriptions of regular power and network outages in formulating design requirements. With respect to the technology baseline of users, the GCN needs assessment found that: About a third of the physicians interviewed access email regularly… whereas another third claimed that they had at some point accessed email regularly. [1] Thus, it was expected that only a minority of doctors would actually need to acquire computer literacy. B. ML The ML researcher made optimistic assumptions about mobile phone connectivity among its user base and pessimistic assumptions about PC usage. As with GCN, connectivity was assessed qualitatively, drawing on experiences during the four-day trip to the North wherein calls from a OneTouch phone were compared with calls from another provider. While there were occasions of calls dropped or text messages delayed, ML concluded that the quality was sufficient for most basic uses. With regards to computer literacy, ML concluded that: The daily routine of a physician in Ghana does not revolve around computers. I can’t go a day as a physician [in the U.S.]... without using a computer – taking orders to checking lab values – but in Ghana it’s all people ordered, from medical records to orders, operative notes… Everything is done by paper! [2] These observations are actually consistent with those of GCN, but what gives these findings an added dimension is the ability of the ML researcher to make a direct comparison of this


Figure 7. Urban Hospital (DSL), Wed-Tue

Figure 8. Teaching Hospital (VSAT), Thu-Wed

Figure 9. Rural Hospital (VSAT), Mon-Thu

Packet Loss Rates (%) and Round-Trip Average (ms.) for three hospital servers in Ghana. Left y axis shows median, 5th and 95th percentile of RTA.

environment with his experience in a leading hospital in the U.S. Thus, while GCN framed its findings as “most doctors do indeed have some experience with computers” and would use them more often given better access, ML concluded that “computers are not part of the daily routine of a Ghanaian physician” and so are not a useful option. C. Discussion Part of the difficulty in producing consistent assessments of conditions on the ground is that while metrics such as the frequency of power outages and the availability of DSL can be measured and mapped in great detail, it is up to each investigator to determine exactly what metrics need to be measured, how rigorous the measurement needs to be, and what quality of service constitutes something usable by the target community for the specific application. GCN had no resources or expertise with which to improve the mobile phone infrastructure, so it focused more on network infrastructure. ML had no expertise with network infrastructure, but instead saw an opportunity to address a very significant, non-technical barrier in mobile phone usage. A closer look at the details of connectivity suggests that both projects were somewhat optimistic in downplaying the infrastructural limitations. The qualitative assessments verified expected connectivity barriers, whereas additional issues remained hidden.

GCN anticipated power outages and network disconnections occurring several times a day, but the regularity with which (nominally acceptable) bandwidth was unusably low was a surprise. For example, certain pieces of systems software on the local servers needed to be configured with estimates of the worst-case time needed to transfer an 8KB file chunk; the initial estimate of 20 seconds (~3.3Kb/s) was eventually increased to 5 minutes (~0.2Kb/s). Connectivity data obtained after the deployment of GCN show that the difference between best- and worst-case performance can be extreme. Figures 7, 8 and 9 illustrate the volatility of network performance at three sample hospitals. Figure 7 shows a DSL network connection with moderately variable packet loss rates and round trip times. However, Figure 8 illustrates the case of a satellite connection so overloaded that, on weekdays between 10:00 AM and 4:00 PM, packet roundtrip times to the public GCN server in Ghana consistently exceeded 10 seconds and loading google.com took 15 minutes. While the GCN software was successfully able to transmit doctors’ messages when congestion decreased in the evening, a doctor who requested a consultation in the morning would wait until the following day to receive a response (even if the consultant responded immediately upon receipt of a message). (The irony here is that this hospital, with the largest number of dedicated IT staff - 6 people - thus experienced the most apparent ‘outages’.) In contrast, Figure 9 shows the characteristics of a satellite connection that worked reliably and consistently. As ML is not based on Internet connectivity, an analogous examination of ML’s assessment of mobile phone coverage would require assessing the quality of service of OneTouch voice calls and text messages. While we do not have this information, we can look instead at geographical coverage (Figure 10). While OneTouch has more coverage than any other provider in Ghana, the coverage map shows that vast regions of the country remain out of coverage area, leaving doctors working in those regions at a significant disadvantage. Clearly, a trip to a single urban center in the North does not systematically gauge the limitations of coverage throughout the country, let alone quality of service within current coverage areas. To be clear, the issues illustrated above have not been a cause of project failure in either case. Further, exhaustive

Figure 10. OneTouch GSM Coverage Areas. (Source: GSM Association)


quantitative assessment of all such issues in advance of deployment is not always cost-effective or even possible (for example, one cannot measure DSL links that are not installed). What we suggest here is that care must be taken to prevent “blind spots” in assessing infrastructure and that such blind spots can arise from the investigators’ backgrounds. VI. SOLUTION USAGE AND APPROPRIATION In this section, we discuss the usage of the technological solutions that arose from each project as well as the ways in which the solutions were appropriated for uses other than those they were intended to enable. We then discuss the role of what we call technological specificity in these processes. A. GCN As described previously, GCN adoption has been relatively slow; what is notable is that GCN experienced a better rate of adoption in the smaller hospitals than in the large ones. At the smallest deployment hospital, with only 2 doctors on staff, one of the doctors continues to log into the system and post cases on a roughly bi-weekly basis. In the mid-sized hospitals, the response was mixed, with a handful of doctors using the system every week or two but the majority losing interest. At the largest deployment hospital, however, after the initial flurry of activity surrounding the presentations and training, none of the doctors continued to use it on a consistent basis. Follow-up interviews shed some light on this disparity. One issue was computer access. In the smallest district hospital, the doctors shared one computer, but since there were only two doctors, having access to the computer was never a problem. In contrast, access to computers was an issue at the larger hospitals. One doctor said: Sometimes you go to the library, you see someone at the computer for 15 minutes; you don’t have that time to waste [waiting for the computer]. Another issue was the match between the system’s use case and the needs of the pilot participants. Through its design and pool of GPSF consultants, GCN had targeted general practitioners (GPs) in the North who wished to consult with urban and overseas medical specialists. This matched the needs of district hospital doctors in the South who had no ready access to specialist consultation and saw great value in the ability to connect with specialists (either in Ghana or overseas). The popularity of this system in the district hospitals is a very promising precedent for future deployments in the more rural North. In contrast, physicians at the larger Southern hospitals could speak and consult with other doctors and specialists more easily in-person than over the computer. GCN did not address the needs of urban specialists looking to (In tap into the global community of sub-specialists. evaluation interviews, many of the urban doctors requested a greater number of sub-specialists within the system.) GCN Appropriation Beyond the expected use for consultation, there were unanticipated uses of the system. First, 6 of the 39 ‘conversations’ observed were purely social in nature. This

was surprising in light of how ‘medicalized’ the investigators perceived the interface. Yet perhaps because their means of communicating with remote doctors are so limited, it appears the barriers between profession and person are much more fluid than anticipated. Second, many of the doctors discussed and requested functionality in the system for sharing literature and PowerPoint presentations, to the extent that these features were included in the upgrade from the pilot to the ongoing deployment. This is consistent with the findings mentioned earlier that a larger screen size was important in dealing with more information-intensive tasks, and also provides insight into the kind of tasks matched to a computer’s affordances. B. ML As previously noted, ML experienced an incredible rate of adoption, with 1700 of 2000 members of the GMA signing up within the first four months. While there are no statistics available concerning the fraction of usage that is related to consultation, there is little doubt (judging from the interviews and from multiple instances of observing doctors as they received calls) that ML is used frequently for consultation. There are multiple reasons for this popularity, some of which are hard to distinguish from the affordances and popularity of mobile phones themselves. (One doctor claimed his phone bill dropped from 150 USD per month to 8 USD per month after joining the program.) First, mobile phones are a popular medium for medical consultation because the realtime nature of voice calls is often critical to treating an emergency case – the three doctors who volunteered information on the breakdown of emergency/non-emergency cases reported that around 80% of cases that require further consultation are indeed emergency cases. One said: I prefer phone calls to SMS, because I prefer an immediate answer, and also so I can make sure the phone is on. If I’m dealing with a case right now, I want to know what to do when moving ahead as soon as possible. Second, phone skills are more widespread than computer skills. Because mobile phones are practically ubiquitous in Ghana, there is a lower learning curve as opposed to computers, which are owned by only one in three doctors. A few doctors claimed that texting on a mobile phone was easier than typing on a keyboard. While many of the junior doctors demonstrated great proficiency with both typing and texting – the vast majority sent multiple text messages a day – a few of the senior doctors demonstrated great difficulty typing during training sessions. Third, it promotes more tightly-integrated workflows. One doctor said: I use [ML] a lot and I think it is wonderful. If you want to talk to anybody concerning a case… concerning anything relating to your practice... it gives you a chance to relax and really talk. It’s so good. It’s a wonderful idea. Another said: There has been a move to ban mobile phones in certain hospitals. It is a very very big mistake, because all they are going to realize is that this is actually going to decrease


efficiency rather. Consultation is not going to be working as well as it used to. ML Appropriation ML was originally framed as a system to ensure continuity of care in long-term and referral patients, but after the system had been established for three months the ML investigator noticed its emergent effect in fostering camaraderie in the medical community. He said: I was talking to doctors and they were telling me. “Yeah, I’m reconnecting with classmates.” Indeed, many doctors were up-front about the fact that ML had been a boon, not just for consultation, but for facilitating social interaction within their community. The mobile phone has also integrated itself into administrative and management processes within large hospitals. One doctor said: You don’t have walk down somewhere or you send a patient down... it reduces the whole bureaucratic... pushing around of patients. Personal mobile phones, both on the ML program and not, are regularly used to set up diagnostic tests at the laboratory, to confirm insurance claims forms, and also for the doctor on call during the night to discuss a change in a patient’s care with the doctor who admitted that patient. In short, where a U.S. hospital might use infrastructure such as pagers and site-wide internal communications systems, large Ghanaian hospitals improvised solutions using personal mobile phones. C. Discussion In reflecting upon differences between GCN and ML in usage and appropriation, we will focus on an important distinction that we will call ‘specificity’. Two solutions with different ‘specificity’ can be targeted at exactly the same task and be based on an equally nuanced understanding of workflows and use cases; the difference lies in the types and number of layers of technology which make up the solution and the degree to which they are specific to the solution. For example, the GCN system presents not only a robust asynchronous communication medium, but also an in-hospital server as well as an end-user Web application. ML, in its first phase, focuses exclusively on tackling the cost barrier of existing phones using existing networks. As we have seen, these two projects with similar goals at the start ultimately resulted in two very different usage outcomes – particular in the area of adoption. Investing time and resources in needs assessments and design process, the GCN project produced a highly ‘specific’ system to address not only failures in internet infrastructure but also social network gaps. However, adoption has been slowed by the need to introduce the system incrementally into hospitals. The ML project achieved broader adoption over a much shorter period of time, in part by relying on the existing availability and high adoption levels of mobile phones. Similarly, the lower specificity of the ML solution seems to contribute to a greater range of user appropriation behaviors.

While it is tempting to conclude that providing solutions with lower specificity is strictly more desirable – and in many cases it may be more desirable – it must also be remembered that utility comes in many forms. GCN’s higher degree of specificity is due to multiple factors. First, the social networking application is required to meet the GCN goal of enabling isolated doctors to build social capital in an extended geographic network of colleagues. While ML assumed that the doctors that needed to work together already had each other’s mobile phone numbers, GCN had determined that a large portion of rural, immigrant, and junior doctors did not have a strong network of contacts [20]. Second, GCN’s emphasis on overseas consultants implies a need for lowbandwidth, asynchronous communication as opposed to voice calls. Third, GCN operated as a development project, maintaining statistics of usage metrics in order to facilitate evaluation; for now, ML relies entirely on the built-in reporting mechanisms in the OneTouch network. GCN can map consultations made to specific case outcomes, while OneTouch – which does receive very high praise from enrolled doctors who offered their feedback on the system – cannot. Moreover, as previously mentioned, GCN’s goals include technological innovation as well as social impact. GCN assumes connectivity in rural areas will remain an ongoing challenge, while ML relies on the assumption that, with time, OneTouch coverage will be able to reach even the most remote doctors. This question echoes one of the fundamental tensions in ICTD research: as pragmatists, we aspire towards demonstrating the greatest social impact in real communities today, while as researchers, we try to identify what fundamental limitations exist and how these can be tackled in years to come. It is to be expected that untested technology would experience more hurdles in the short run, while its long-term contribution is yet to be seen. In short, then, a project can easily have a number of goals that might be frustrated by a lower-specificity solution. A final note about lower-specificity solutions concerns their potential to be too widely appropriated. In the smallest district hospital (where GCN experienced greater adoption), both doctors tended to switch off or mute their phones while at the hospital in order to minimize distraction. Even in the larger urban hospitals, half of the 20 doctors who discussed their mobile phone usage claimed to keep their phones off while working because of distraction. Hospital administrators have related concerns; at one teaching hospital, a memorandum was circulated to all the doctors banning the use of mobile phones in many locations …to forestall the negative impact of mobile phone frequencies on medical equipments and improve the work ethics of staff. It is also announced… that it is a serious offence to disconnect life-supporting equipment in order to use their power sources (socket) to charge mobile phones. In view of concerns such as these, we suggest that researchers keep in mind both the (immediate) benefits and (eventual) costs of rapid adoption.


VII. RELATED WORK ICTDs for Healthcare in the Developing World The research on ICTDs to promote healthcare has a long and rich history [15],[34]. In the context of developing regions, remote medical consultation has been a popular and relatively successful approach [32]. Computers [3], mobile phones [25] and a combination of both [27] have demonstrated utility in a variety of settings. Framing the Problem [21] provides a useful overview and categorization of how ICTs are conceptualized from a variety of different fields. Theory on ‘framing the problem’ can be drawn from socialconstructivism, although we are not advocating here for a change from ICTD’s traditional focus on pragmatism and advocacy/participatory research. Much of the work on qualitative methodology emphasizes the importance of reflexivity and self-awareness in order to minimize such bias [8], but the fact remains there are institutional accountabilities which no amount of methodology can shake. We can learn from the example of anthropologists who have challenged their own role in the colonial apparatus [11],[30]. For example, [17] provides an interesting comparison of how different sectors tackle the management of kiosks in India. Partnerships and Deployment ICTD researchers have an important role to play at the crossroads of business administration and government policy, identifying how both contribute to the goal of development and how their contributions interplay. Many examples in ICTD research and development literature study the role of institutional players, although these papers are typically evaluative rather than action-oriented [18],[23],[24]. The literature on partnership selection and cultivation in ICTD is small but growing rapidly [26],[28],[31]. Specificity In reaction to the variable success of many deployments of general ICT, such as a number of telecenter and ‘hole-in-thewall’ computing initiatives, much of the focus of the HCI ICTD research community has turned towards purpose-built technology [4],[33]. Many of those doing systems research focus instead on providing basic computing or connectivity functionality [13],[29]. [6] presents a useful survey of ICTD literature which mentions a pre-1999 trend towards non-

specific technologies such as connectivity and a post-1999 trend towards specific solutions such as software and VOIP. VIII. CONCLUSION In this paper, we have presented parallel case studies of two ICT projects addressing the need for improved medical consultation among doctors in Ghana. We have examined how the two projects have been shaped by the institutional context and the identity of their researchers. We looked at how the partnerships formed affect the solution outcome, delivery, and adoption; we argued that even the task of assessing technological infrastructure is far from objective; and we noted the implications of the ‘specificity’ of each solution. We conclude with an open question raised by these discussions, summarized in Table 3. In ICTD research, much attention has been given to the socio-cultural, political, and economic contexts of target communities – yet ‘difference’ is a measure between communities; it is only by critically examining our own research community that we can understand the influence and impact of communities on each other. ICTD is linked with economic, social, political, and human development agendas [6]. Regardless of whether we take an integrated approach [10] or focus instead on specific local needs [14], there are still institutional, societal, and individual layers at play in our interventions. Researchers are embedded in contexts of existing friendships, collaborations, expertise and agendas, and we need to be conscious of what kind of consequences our decisions to draw on them have on project outcomes. In this paper we have seen impacts on everything from research agendas to infrastructure assumptions, yet the literature on these avenues of choice remain fragmented across a variety of other communities including CSCW and organizational behavior. ICTD researchers need to increase the exchange of ideas between these communities. For example, in the course of establishing GCN, the task of generating a viable ‘business model’ for the technology providers was not addressed, but a social entrepreneurship community would never allow such an omission. ML has had great adoption success, but technical infrastructure for evaluating its impact was neglected. We need a theoretical framework under which to unite the different fields of research for ICTD practitioners, one surrounding not just technology design, but also technology framing, partnership, assumptions, and deployment.

Table 3 – Impacts of project context on project decisions and outcomes By focusing on connectivity in the rural North, GCN targets a minority of doctors with the greatest need. Framing By focusing on airtime cost, ML impacts the majority of doctors, the bulk of whom are in the urban South. Working through MOH allowed GCN to iterate rapidly with many hospitals early on, but with greater overhead later. GCN`s Partnerships and commitment to the U.S.-based GPSF was a key factor that tied it to a computer-based platform instead of mobile phones. Deployment GMA advertised ML effectively while OneTouch quickly took ownership of project execution and maintenance. ML’s approach is highly dependent on the vagaries of a single national operator while GCN’s is at the mercy of individual hospitals. GCN found the networks to be more challenged than anticipated. Infrastructure ML found mobile phone coverage lower than anticipated. GCN`s higher specificity allowed it to incorporate evaluation indicators and address not only physical communication gaps but also Solution gaps in social capital. Usage limited by access to computers in larger hospitals. ML adoption was much higher given the real-time nature of mobile phones and the fact that they were already widely tested and used. Lower specificity facilitated adoption and appropriation while making it difficult to evaluate the program.


[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] Field notes from the GCN investigators’ journals. Phone conversations between the GCN investigators and the ML investigator. K. Brauchli, D. O’Mahony, L. Banach, and M. Oberholzer, “iPath – A Telemedicine Platform to Support Health Providers in Low Resource Settings,” J. IT in Healthcare 3, 4 (2005), 227-235. P. Braund and A. Schwittay. “The Missing Piece: Human-Driven Design and Research in ICT and Development,” Proc. ICTD 2006, IEEE (2006), 2-10. E Brewer, M. Demmer, B.W. Du, M. Ho, M. Kam, S. Nedevschi, J. Pal, R. Patra, S. Surana, and K. Fall. “The Case for Technology for Developing Regions,” IEEE Computer 38, 6 (2005), 25-38. J. Bussell. “International Norms on ICTs for Development: New Data, Initial Findings, and Opportunities for Analysis,” American Political Science Association Annual Meeting, APSA (2005). V. Cerf, S. Burleigh. A. Hooke, L. Torgerson, R. Durst, K. Scott, K. Fall and H. Weiss. Delay-Tolerant Networking Architecture, RFC 4838, IETF, 2007. J.W. Creswell and V.L. Plano Clark. Designing and Conducting Mixed Methods Research. SAGE, Thousand Oaks, CA, 2007. M. Demmer. A Delay Tolerant Networking and System Architecture for Developing Regions. Ph.D. dissertation, Univ. of California, Berkeley, CA, 2008. R. Duncombe and R. Heeks. “Information, ICTs and Small Enterprise: Findings from Botswana,” In H. Katrak and R. Strange (eds.), SmallScale Enterprises in Developing and Transitional Economies. Palgrave, Basingstoke, UK, 285-304. P. Dourish. “Implications for Design,” Proc. CHI 2006, ACM (2006), 541-550. K. Fall. “A Delay-Tolerant Network Architecture for Challenged Internets,” Proc. SIGCOMM 2003, ACM (2003), 27-34. R. Fonseca and J. Pal. “Computing Devices for All: Creating and Selling the Low-Cost Computer,” Proc. ICTD 2006, IEEE (2006), 11-20. R. Gerster and S. Zimmerman. “ICTs for Poverty Reduction,” Information for Development Online, Jul-Aug 2003. http://www.i4donline.net/issue/july03/pdfs/icts_poverty_reduction.pdf W.R. Hersh, D.H. Hickam, S.M. Severance, T.L. Dana, K.P. Krages, and M. Helfand. “Diagnosis, Access and Outcomes: Update of a Systematic Review of Telemedicine Services,” J. Telemed. Telecare 12, S2 (2006), 3-31. D. Kuo, D.R. Gifford, and M.D. Stein. “Curbside Consultation Practices and Attitudes Among Primary Care Physicians and Medical Subspecialists,” JAMA 280, 10 (1998), 905-909. R. Kuriyan and J. Bussell. “Understanding Concepts of ‘Development’ and Linkages to Deployment Strategies in ICT4D in India,” in A. Palackal and W. Shrum (eds.), Information Society and Development: The Kerala Experience. Rawat, New Delhi, 2007.

[11] [12] [13] [14] [15]

[16] [17]

[18] R. Kuriyan and I. Ray. “Public Private Partnerships and Information Technologies for Development in India,” Proc. ICTD 2007, IEEE/ACM (2007), 310-320. [19] B. Latour. Science in Action. Harvard Univ. Press, Cambridge, MA, 1987. [20] R. Luk, M. Ho and P.M. Aoki. “Asynchronous Remote Medical Consultation for Ghana,” Proc. CHI 2008, ACM (2008), 743-752. [21] W. Orlikowski and C.S. Iacano. “Desperately Seeking “IT” in IT Research – A Call to Theorizing the IT Artifact,” Information Systems Research 12 (2001), 121-134. [22] R. Patra, S. Nedevschi, S. Surana, A. Sheth, L. Subramanian and E. Brewer. “WiLDNet: Design and Implementation of High Performance WiFi Based Long Distance Networks,” Proc. NSDI 2007, USENIX (2007), 87-100. [23] L.R. Peattie, Rethinking Ciudad Guyana. Univ. of Michigan Press, Ann Arbor, MI, 1987. [24] E. Peter. “The Challenges of the Institutional Turn: New Interdisciplinary Opportunities in Development Theory,” in V. Nee and R. Swedberg (eds.), The Economic Sociology of Capitalism. Princeton Univ. Press, Princeton, NJ, 2005. [25] K. Phipps, G. Sangudi, and S. Woolway. What Works: HealthNet Uganda’s Evolution from NGO to Sustainable Enterprise: Portable Healthcare Service Delivery to Uganda’s Rural Areas. World Resources Institute, 2003. [26] D. Ramachandran, M. Kam, J. Chiu, J. Canny, and J.F. Frankel. “Social Dynamics of Early Stage Co-Design in Developing Regions,” Proc. CHI 2007, ACM (2007), 1087-1096. [27] J. Royall. “SatelLife – Linking Information and People: The Last Ten Centimeters,” Development in Practice 8, 1 (1998), 85-90(6). [28] Y. Schwartzman and T. Parikh. “Establishing Relationships for Designing Rural Information Systems,” Extended Abstracts, CHI 2007, ACM (2007), 1845-1850. [29] A. Seth, D. Kroeker, M. Zaharia, S. Guo, and S. Keshav. “Low-Cost Communication for Rural Internet Kiosks Using Mechanical Backhaul,” Proc. Mobicom 2006, ACM (2006), 334-345. [30] L. Suchman. “Practice-Based Design of Information Systems: Notes from the Hyperdeveloped World,” The Information Society 18, 139-144. [31] S. Surana, R. Patra, S. Nedevschi, and E. Brewer. “Deploying a Rural Wireless Telemedicine System: Experiences in Sustainability,” IEEE Computer 41, 6 (2008), 48-56. [32] D.J. Vassallo, P. Swinfen, R. Swinfen, and R. Wootton, “Experience with a Low-Cost Telemedicine System in Three Developing Countries,” J. Telemed. Telecare 7, S1 (2001), 56-58. [33] M. Warschauer. Technology and Social Inclusion. MIT Press, Cambridge, MA, 2003. [34] P. Whitten and B.D. Sypher. “Evolution of Telemedicine from an Applied Communication Perspective in the United States,” Telemedicine and e-Health 12, 5 (2006), 590-600.


Improving Child Literacy in Africa: Experiments with an Automated Reading Tutor
G. Ayorkor Mills-Tettey, Jack Mostow, M. Bernardine Dias, Tracy Morrison Sweet, Sarah M. Belousov, M. Frederick Dias, Haijun Gong

Abstract—This paper describes a research endeavor aimed at exploring the role that technology can play in improving child literacy in developing communities. An initial pilot study and subsequent four-month-long controlled field study in Ghana investigated the viability and effectiveness of an automated reading tutor in helping urban children enhance their reading skills in English. In addition to quantitative data suggesting that automated tutoring can be useful for some children in this setting, these studies and an additional preliminary pilot study in Zambia yielded useful qualitative observations regarding the feasibility of applying technology solutions to the challenge of enhancing child literacy in developing communities. This paper presents the findings, observations and lessons learned from the field studies. Index Terms—Developing Regions, Education, Educational Technology, Literacy

is a key part of the global development agenda. It is a complex concept with multiple definitions. An international expert meeting at UNESCO in 2003 proposed the following definition: “Literacy is the ability to identify, understand, interpret, create, communicate and compute, using printed and written materials associated with varying contexts. Literacy involves a continuum of learning in enabling individuals to achieve their goals, to develop their knowledge and potential, and to participate fully in their community and wider society.” [22] The United Nations recognizes literacy as a human right, noting that basic education, of which literacy is the key learning tool, was recognized as a human right over 50 years ago in the Universal Declaration of Human Rights.
ITERACY The research reported here was supported in part by the discretionary gifts to the TechBridgeWorld research group at Carnegie Mellon University, by the Qatar Foundation for Education, Science, and Community Development, by the National Science Foundation under ITR/IERI Grant No. REC-0326153, by the Institute of Education Sciences, U.S. Department of Education through Grant R305A080628 to Carnegie Mellon University, and by the Heinz Endowments. The opinions expressed are those of the authors and do not necessarily represent the views of any of our sponsors. G. A. Mills-Tettey is with the Robotics Institute at Carnegie Mellon University, Pittsburgh, PA 15213 (phone: 1-412-268-8645; fax: 1-412-2686436; email: ayorkor@cmu.edu). J. Mostow, M. B. Dias, S. Belousov, and M. F. Dias are also with the Robotics Institute at Carnegie Mellon University, Pittsburgh, PA 15213 (email: mostow@cs.cmu.edu, mbdias@ri.cmu.edu, sarahtbw@ri.cmu.edu, mfdias@ri.cmu.edu). T. M Sweet and H. Gong are with the Statistics Department at Carnegie Mellon University, Pittsburgh, PA 15213 (email: tsweet@andrew.cmu.edu, haijung@andrew.cmu.edu)



As a contribution to the discourse on applying information and communication technologies (ICTs) to address development challenges, we describe a study exploring the potential role of computing technology in improving English literacy among Ghanaian and Zambian children who attend school in English but have low reading achievement levels. Low functional literacy among individuals who have completed primary school is not an uncommon problem in developing communities. For example, UNESCO reports that in 2000, more than 1 in 3 adults with a fifth-grade education in Chad and Niger reported that they could not read [21]. In other cases, individuals may finish primary school reading below the expected level. In a representative sample of Ghanaian public schools, reading achievement levels measured by the government-administered Criterion Referenced Test in 2000 indicated that fewer than 10% of the children in grade six were able to read with grade level mastery [9]. Several factors contribute to this problem. For the average child from a rural or low-income urban background in Africa, reading is not part of daily family life, and sometimes parents are not themselves literate. In Ghana and Zambia specifically, most children speak one of a number of local languages at home but attend a school taught in English, the official language for both countries. Typically, under-resourced schools with overcrowded classrooms offer few opportunities for individual attention while developing reading skills. The project described in this paper is a proof-of-concept study to investigate whether an automated computer-based reading tutor that provides guided reading practice can significantly improve the reading proficiency of children in a developing community, even if they have no prior familiarity with computers. We focus our study on children in Accra, Ghana, and Mongu, Zambia. As illustrated in Figure 1, we began by employing the Reading Tutor in a preliminary three-week-long pilot study in Accra, Ghana. The pilot study was used to explore technical and operational feasibility and to motivate partnerships and funding for a longer term study. We followed up with a fourmonth-long controlled study in Accra, in collaboration with the Ghana-India Kofi Annan Centre for Excellence in ICT (AITI-KACE) and with input from Associates for Change, an


educational research firm in Accra. The controlled study aimed at quantitatively measuring the educational effectiveness of the tutor with children in Accra. Observations and lessons from these experiences fed into another pilot study in Mongu, Zambia, with the plan of following up with an additional controlled study in Zambia in the future.
Exploring the role of technology in improving English Literacy in Africa Existing tool: Automated Reading Tutor for guided oral reading practice Ghana June 2005 Pilot study • Gauge children’s reactions to Reading Tutor • Explore out-of-school usage model • Assess feasibility of test instruments May-Sept 2007 4-month controlled study • Measure reading progress • Compare progress across 3 socio-economic groups • Assess sustainability of outof-school usage model July 2008 Pilot study • Gauge children’s & teachers’ reactions to Reading Tutor • Explore in-school usage model • Explore government literacy requirements and tests Future work Year-long controlled study • Measure reading progress • Compare to nontechnological approach to guided reading practice • Assess sustainability of inschool usage model Proposed future work Zambia New tools

Reported work

Figure 1 - Project outline

Section II describes the skills that are involved in reading and introduces the automated tutor we used in this project. Sections III, IV, and V describe the pilot field study in Ghana, the 4-month-long controlled study in Ghana, and the pilot study in Zambia respectively. Section VI distills practical lessons learned in the implementation of these field studies and Section VII concludes with a discussion of future work. II. BACKGROUND As a cognitive proficiency, reading involves several component skills such as phonemic awareness, decoding, fluency, vocabulary, and comprehension [14]. Phonemic awareness is the ability to perceive individual sounds or phonemes in words. Building on this awareness, a child learns the alphabetic principle that spelling generally maps systematically to pronunciation. The child also learns specific letter-sound correspondences and the ability to correctly pronounce written words (decoding). Fluency is the ability to read text accurately, quickly, and expressively and is an important foundation for comprehension. A rich vocabulary is essential for comprehension and effective communication. Children develop these skills through a variety of experiences, including skilled instruction. Research has shown that regular guided oral reading plays an important role in developing

reading skills, particularly fluency and comprehension [14]. Such guided oral reading may happen in small groups in a classroom setting, or with parents at home. Technology may also assist in this process: Scientific Learning’s Reading Assistant [1] and Project LISTEN’s Reading Tutor [11] are examples of computer-based tools that use automated speech recognition to provide a guided reading experience for the user. Project LISTEN’s Reading Tutor has demonstrated success in improving the reading ability both of children whose first language is English [11][12][13] and of children learning English as a second language (ESL) in the United States [16] and in Canada [18]. The project described in this paper used Project LISTEN’s Reading Tutor, developed at Carnegie Mellon University. The Reading Tutor displays stories on a screen and “listens” to a child read aloud. By using speech recognition to analyze the child’s reading, the Reading Tutor is able to give graphical and spoken feedback. It gives help when it detects a long pause, a severely misread word or a skipped word, and also when the reader clicks for help. The tutor may speak the whole word out loud or decompose the word into syllables or phonemes and speak out each part while highlighting it. It may also give a “rhymes with” hint, or read the sentence by playing a fluent human narration to model expressive reading. The Reading Tutor includes a wide variety of stories at different reading levels. It takes turns with the child in selecting a story to read. It monitors the child’s reading progress and selects stories at an appropriate level for the child. For readers at early stages of development, the Reading Tutor also includes word-building exercises to develop knowledge of spelling-to-sound correspondences. Videos of the Reading Tutor in use may be found at Project LISTEN’s website [17]. III. GHANA PILOT STUDY A. Goal The 2005 Ghana pilot study aimed to evaluate the practicality of a technological approach to guided reading practice in Accra, and investigate the feasibility of conducting a longer term controlled study. Specifically, we wished to answer the following questions: 1. Partners and logistics: How feasible is it to engage partners and arrange the logistics for such a study? 2. Learning to use the Reading Tutor: How quickly do children with no prior computer experience learn to operate the Reading Tutor, and what instruction do they need in order to do so? 3. Speech: Does the speech recognition software perform acceptably with Ghanaian accents, and can the students understand the narrated speech? 4. Tutor content: Do the children find the reading material in the tutor engaging? 5. Usage sessions: What is an effective length for a tutoring session?



Reading measures: Which test instruments can be used easily and effectively to assess reading proficiency in the Ghanaian setting?

B. Participants and Methodology We chose to focus on the needs of children from lowincome families attending public school because they have a high risk of low achievement in reading. In consultation with Associates for Change, we targeted children in grades two through four since this is a key period for developing reading skills after the initial adjustment to the primary school environment. We restricted our work to an urban environment where computers are more readily available, but where significant literacy challenges still exist. We employed an out-of-school usage model in which practice with the Reading Tutor was supplemental to regular school activities. The pilot study involved qualitative observations of children as they used the Reading Tutor. The study was conducted by the first author who is a native of Ghana. She was assisted by a local volunteer. Two groups of children participated in this study. One group comprised twelve children in grades two through four from an under-resourced public school. They used the Reading Tutor at an internet café near their school for 20-30 minutes each day over a threeweek period. The other group comprised six children from a mixed low- and middle-income neighborhood. They used the tutor on laptops in the home of the researcher, for 20-30 minutes each day, three days a week over the same period. C. Results and Observations Partners and Logistics: The school, internet café, and the parents of all participating children agreed without hesitation to participate in the project, and logistics were arranged very quickly. The internet café donated time on four desktop computers; thus, four children could use the Reading Tutor at a time. We engaged the remaining children in other activities while they waited their turn. This worked fairly well for the pilot project, but would not be a feasible approach for a longer study involving a larger number of children. Learning to use the Reading Tutor: Although one child had previously played a computer game, none of the other children had used a computer before. The children were given an initial 10-15 minute hands-on lesson introducing them to the computer and showing them how to use the mouse, keyboard and software. The children had trouble understanding the Reading Tutor’s built-in automated tutorials on how to operate it and use the keyboard, perhaps due to the unfamiliar narration accent or the use of words such as “roster” that are common in American but not Ghanaian English. This difference in language use is an example of the need to localize tools for a given setting. We explained the tutorials one-on-one to the children during their first two sessions. Some of the questions the children asked during the first few sessions were on keyboard use and on navigating the tutor. By their second or third session, most of the children were able to operate the Reading Tutor without help.

Speech: The speech recognition capability appeared to work adequately with the children’s accents: it accepted a great majority of words that were read correctly. Also, based on the graphical feedback given by the tutor, the children quickly learned to recognize when it did not “hear” them correctly and to repeat themselves when necessary. After the initial difficulty with the built-in tutorials, the children appeared to understand the prompts given by the Reading Tutor. Tutor content: Overall, the children were very enthusiastic about using the Reading Tutor. Most of them appeared to enjoy the world-building exercises and the stories in the tutor. However, we noticed that a couple of children who were older than the norm (11 or 12 years old) but had a kindergarten or first grade reading level were not fully engaged by the content of the simple stories, such as “Sam sat on a mat,” available for their reading level. There was a discrepancy between these children’s maturity and their reading ability. Furthermore, we believe it would be good to incorporate more local content, such as Ghanaian folk tales, into the tutor. Usage sessions: We noticed that the better readers could use the tutor for more than half an hour at a time without getting bored, whereas the more challenged readers would tire after about twenty minutes, but would still look forward to their next turn. We thus decided to limit the length of usage sessions in the controlled study to 20-30 minutes. Reading measures: Although the pilot study did not aim to quantitatively measure reading progress, we identified and tested reading measures that we would use in the controlled study. The instruments we chose to use to assess reading ability were an oral reading fluency test [5] and the Test of Written Spelling (TWS) [8], both of which are hand-scored tests that have been frequently used in previous studies involving the Reading Tutor, and are psychometrically reliable, fast and easy to administer. The fluency test is timed reading exercise in which the child is given a gradeappropriate story to read. It is scored as the number of words read correctly in one minute. Fluency is essential to comprehension and is a sensitive measure of growth in proficiency [19]. The TWS is a dictation exercise in which the child writes down words read aloud by the tester in order of increasing difficulty. The test ends once the child has incorrectly spelled five words in a row, at which point it is assumed that the difficulty of the remaining words exceeds the child’s proficiency. This allows the same test to be used for multiple grade levels. TWS is scored as the number of words spelled correctly. Spelling is like decoding in that it tests printto-sound mappings but it is easier to assess reliably. We tried out the fluency test and the TWS with the pilot study participants, and they did not have trouble with the format of either test. To combat their nervousness about being examined, we avoided using the term “test.” We instead explained that these exercises were to help figure out how the Reading Tutor could assist them and we reassured the children that they would not be given a grade. We modified the passages for the fluency test to use Ghanaian names rather


than American names, to make them more recognizable to the students. The children were familiar with the dictation format of the TWS, but again were unfamiliar with some words such as “tardy” which are not regularly used in Ghanaian English.

gender boundaries into two groups, Tutor-1st and Control-1st, as shown in Table I. We used a two-treatment crossover study design: for the first half of the study, the Tutor-1st group used the Reading Tutor while the Control-1st group had no additional reading intervention, and for the second half, the roles were switched: the Control-1st group used the Reading Tutor while the Tutor-1st group had no additional reading intervention. Each half of the study lasted nine weeks (two months) during which there were daily usage sessions of approximately half an hour per child, although attendance and usage varied considerably. This crossover study design, illustrated in Table II, had the advantage of equity in the sense that all participating children had the opportunity to use the reading tutor. It also avoided having to pair similar children for comparison since each child essentially served as his or her own control. The potential disadvantage, which was that carryover treatment effects from the first half may have affected the second half, was mitigated by having half the children participate first in the control, while the others participated first in the treatment group.
Table I - Study participants by school, grade and gender Number of Children Tutor-1st Control-1st School Grade Female Male Female Male S1 Grade 2 3 2 2 3 (29 children ) Grade 3 5 2 1 3 Grade 4 1 3 3 1 S2 Grade 2 2 3 3 2 (30 children) Grade 3 3 2 1 4 Grade 4 3 2 3 2 S3 Intermediate 3 4 3 5 (30 children) Advanced 3 5 5 2

Figure 2 - A child in Accra reading a story using the Reading Tutor

IV. GHANA CONTROLLED STUDY A. Goal Following on the successful pilot study, the goal of the controlled study in Ghana was to quantitatively measure the efficacy of the Reading Tutor in helping children improve their reading skills. Specifically, we wished to determine: 1. Does regular use of the Reading Tutor improve oral reading fluency and spelling? 2. Do treatment effects depend on other factors such as school/socio-economic background, gender, or grade level? In addition, we wanted to learn about the operational sustainability of an out-of-school usage model, taking into consideration installation and maintenance of the software, training of staff responsible for the day-to-day running of the project, transportation of the children between the school and the project site, and other logistics. The controlled study was deployed by project staff at AITI-KACE, with remote training and support by our team in Pittsburgh. B. Study Design / Methodology The controlled study involved eighty-nine children from three schools, representing three socio-economic backgrounds. The participating schools, recruited by Associates for Change, were S1: a private school in a middleincome community, S2: a public school in a low-income community, and S3: an informal educational program for highly disadvantaged children who have never attended formal school. The study involved children in grades two through four of S1 and S2, and in the “Intermediate” and “Advanced” levels (roughly corresponding to grades two and three respectively) of S3. The children were split randomly across school, grade and

A battery of three fluency tests and one TWS were administered to all the children at the beginning of the study (pre-testing), between the two halves (mid-way testing), and at the end (post-testing). The testing was conducted at the children’s schools by AITI-KACE project staff. Three fluency tests were used to better estimate reading ability. The passages used for these tests correspond roughly to first, second, and third grade reading levels respectively, since a test close to the child’s current reading level should give a more sensitive measure of progress. We found that the scores on the three passages were highly correlated, and so for analysis purposes these scores were combined into a single mean fluency score.
Table II – Cross-over Study Design Control-1st Group Tutor-1st Group Pre-testing of all children First half of cross-over Reading Tutor Control: no special (1st nine weeks) intervention Mid-way testing of all children Control: no special Reading tutor Second half of cross-over intervention (2nd nine weeks) Post-testing of all children

The study participants used the Reading Tutor in a


computer lab at AITI-KACE because it had equipment and technical capability that their schools did not. The computers used for the project had 2.4GHz Pentium IV processors, 256MB of RAM, and 16GB hard drives. C. Deviation from Study Design In a normal crossover study, experimental conditions are held constant over the course of the study. However, due to logistical challenges that delayed the start of the project, the first half of this study took place while the students were attending school, whereas the second half overlapped with the school vacation. Because of the different conditions in place during the two halves of the study, we analyzed the results as two different experiments rather than as a single crossover study. The first experiment measured the effect of the Reading Tutor while the children attended school (and thus attended English class as usual). The second experiment measured the effect of the tutor while the children did not attend school (and so did not attend English class). This modified study design is illustrated in Table III.
Table III – Modified Study Design Control-1st Group Tutor-1st Group Pre-testing of all children School + Control: school only Reading Tutor Mid-way testing of all children Control: No school No school + Reading tutor Post-testing of all children

schools) require an adjustment of significance levels [10]. We also compute a measure of effect size, that is, the magnitude of the treatment effect. The measure we use for effect size is Cohen’s d: the difference between group mean gains divided by the within-group pooled standard deviation [3]. The effect size is computed for each school because the treatment effect depends on the school. An effect size of 0.2 is generally considered small, 0.5 medium, and 0.8 large [4].
Fluency pre-test scores by school and grade 180 160 Mean fluency score 140 120 100 80 60 40 20 0 S1 S2 School S3

Grade 2 Grade 3 Grade 4

Figure 3 - Fluency pre-test scores by school and grade
TWS pre-test scores by school and grade
40 35 30 TWS score 25 20 15 10 5 0 S1 S2 School S3 Grade 2 Grade 3 Grade 4

Experiment 1 (1st nine weeks) Experiment 2 (2nd nine weeks)

D. Results and Analysis Figures 3 and 4 illustrate the pre-test scores of children at the three schools, representing their reading proficiency going into the study. It is clear that the S1 children had much higher levels of reading achievement than the S2 children who in turn had higher levels than the S3 children. Pre-testing for our study occurred about two months before the end of the school year. To provide some context for these scores, Table IV compares the fluency pre-test scores with end-of-year norms from schools in the United States [6]. By this standard, the average reading proficiency of the S1 students appears to be at or above the U.S. average whereas that of the S2 and S3 children is lower. Norms from schools in Ghana are not available for comparison. In analyzing each experiment, we focus on gains in fluency and spelling test scores. For experiment 1, the gain is the difference between pre- and mid-test scores; for experiment 2, the gain is the difference between the mid- and post-test scores. A positive gain indicates improvement in the child’s reading proficiency. In each experiment, we use a standard statistical t-test to compare the mean gains of the treatment and the control group. This test yields a p-value indicating how significant the difference is between the means of the two groups. For this analysis we consider p-values of less than 0.016 to indicate statistical significance. This value is smaller and thus more conservative than the commonly used threshold of 0.05 because multiple comparisons (due to the different

Figure 4 - TWS pre-test scores by school and grade Table IV - Fluency pre-test scores compared with U.S. norms Mean pre-test fluency (standard deviation) Year-end U.S. Norm [8] S1 S2 S3 (50th percentile) 2 / Int. 82.4 (44.8) 32.3 (25.9) 11.1 (8.5) 89 3 / Adv 123.3 (31.4) 45.8 (37.4) 20.4 (20.8) 107 4 132.7 (32.7) 59.8 (41.5) -123
Grade / Level

Experiment 1 results by school: Table V shows the results and t-test analysis for experiment 1, when the children were attending school. The reading proficiency of the S2 children who used the Reading Tutor (the “treatment” group) improved significantly more than those who did not (the “control” group), as evidenced by larger gains in both fluency and TWS. The S3 treatment group significantly out-gained the control group in fluency but not in spelling, although there was a positive trend. Finally, there was no significant difference in gains between the S1 treatment and control groups, for either test. Omitted from this analysis are five S1 children who were not present for the midway testing. Due to the small numbers, we did not break down the data to examine


gains for each grade level within each school.
Table V - Comparison of treatment and control gains for experiment 1
Treatment (Tutor-1st group, School N=41) Control (Control-1st group, N=42) t-test comparison of treatment & control gains p-value 0.0833 <0.0001 0.0054 0.2997 0.0039 0.0648 Effect size -0.775 1.731 1.144 -0.457 1.153 0.719


Mean Gain (SD) Fluency S1 8.6 (20.5) 61.4 (22.9) 13.4 (9.3) -1.9 (4.9) 3.9 (2.7) 2.8 (4.0) 21.5 (11.7) 23.4 (21.0) 5.1 (4.4) -0.2 (2.6) 0.3 (3.4) 0.6 (1.7)

(# words read S2 correctly per min) S3 TWS (# words spelt correctly) S1 S2 S3

E. Discussion Effectiveness of the Reading Tutor: The results provide evidence that during the school term, the S2 students (and to a lesser extent the S3 students) who used the tutor gained considerably more than those who did not use the tutor. Thus, the Reading Tutor was helpful for the S2 children. This is a positive outcome since the S2 group most closely represents our target population of children from low-income families attending public school. The proficiency of the S1 children did not appear to be influenced by reading practice with the tutor. A possible explanation for this is that the S1 children might not have had much room to benefit from the tutor since they were fluent readers going into the study. Loss in proficiency over vacation period: Of the statistically significant results highlighted in Tables VI and VII, the negative gains of the S2 children in experiment 2 stand out. A possible explanation for this might be found in studies in the United States which have documented that reading achievement test scores for children from low-income families deteriorate significantly over the summer vacation (a phenomenon referred to as the “summer reading setback”) whereas those for children from middle-income families remain steady or increase slightly [2]. This trend has been attributed to the discrepancy in the reading opportunities and materials available to these two groups over the vacation period. In light of these studies, it is interesting to note that for the S2 children, those who used the Reading Tutor did not deteriorate in reading ability as much as those who did not use it. However, the sessions with the Reading Tutor (totaling, on average, 11.5 hours of reading per child), without English class at school, were not enough to prevent negative gains. Effect of pre-test scores: A complicating factor in the data analysis is the observed unequal average pre-test scores of the two groups, Tutor-1st and Control-1st, despite supposed random assignment of children to the two groups. Table VII shows that this disparity is statistically significant for the S2 children. Higher pre-test scores can be the cause of greater gains [20], and so to determine whether the greater gains of the Tutor-1st group in experiment 1 were due to the Reading Tutor or the higher pre-test scores, we computed the correlation between pre-test scores and gains. We found no significant positive correlation between pre-test scores and gains: the correlation coefficient of mean fluency gains vs. mean fluency pre-test scores was −0.301 (a small negative correlation), and the correlation coefficient of TWS gains to TWS pre-test scores was 0.005 (no correlation). This suggests that the greater gains of the Tutor-1st group in experiment 1 are indeed attributable to the Reading Tutor rather than to their higher pre-test scores.

Experiment 2 results by school: Table VI shows the results for experiment 2, when the children were not attending school. Interestingly, there were negative fluency gains for the S2 children over the vacation, and these were more dramatic for the control group than for the treatment group. We discuss this observation in the Discussion subsection that follows. For the S2 children, the difference in TWS test score gains between the treatment and the control group was not statistically significant. Finally, there was no significant difference between the treatment and the control group in the other two schools for either mean fluency or TWS. Omitted from the analysis are two S1, two S2 and ten S3 children who were absent for testing.
Table VI - Comparison of treatment and control gains for experiment 2
Treatment (Control-1st group, School N=33) Control (Tutor-1st group, N=37) t-test comparison of treatment & control gains p-value 0.613 <0.0001 0.1321 0.6431 0.858 0.1321 Effect size 0.222 1.816 -0.665 -0.197 -0.068 -0.665


Mean Gain (SD) Fluency S1 0.3 (13.4) -9.1 (17.1) 4.8 (6.0) 2.0 (4.1) 0.4 (2.0) 4.8 (6.0) -3.8 (22.6) -39.8 (16.8) 17.8 (27.0) 2.9 (5.1) 0.6 (3.0) 17.8 (27.0)

(# words read S2 correctly per min) S3 TWS (# words spelt correctly) S1 S2 S3

Effect of Gender: In addition to exploring how test score gains varied by school, we examined the data for evidence that test score gains were affected by gender. We focused this analysis on experiment 1 since that data was more expressive. A t-test comparison of the mean gains of females to that of males in each experiment resulted in large p-values (> 0.4), indicating that the mean gains of females and those of males were not significantly different from each other. An Analysis of Variance (ANOVA) also confirmed that gender did not have a significant effect on TWS or fluency gains in experiment 1. Details of these analyses are omitted for brevity.

Table VII - Comparison of pre-test scores for the Tutor-1st group and the Control-1st group Tutor-1st Control-1st (N=41) (N=42) Test School Mean preMean pretest score test score S1 122.0 (45.9) 99.2 (33.9) Fluency S2 65.8 (33.9) 26.2 (26.9) (# words read S3 18.6 (13.2) 12.1 (19.1) correctly per min) S1 24.7 (9.7) 24.0 (9.6) TWS S2 11.1 (7.0) 4.3 (3.7) (# words spelt S3 2.1 (2.2) 1.9 (3.1) correctly)

2. 3.
p-value 0.137 0.0015 0.308 0.850 0.003 0.840

4. 5.

Training Teachers: How long does it take to train teachers to guide students in the use of the tutor? Children’s response: How do the children respond to the Reading Tutor? Test instruments and Reading Tutor content: What test instruments and Reading Tutor content are appropriate for the Zambian setting? Feasibility of long term study: Would a remote partnership between the school and our research group be a feasible model for a longer-term controlled study?

Tutor usage: Another significant difference between the two experiments is that tutor usage in experiment 2 was much lower than in experiment 1, due to a higher level of absenteeism from the study during the school vacation. As Table VIII shows, this is particularly true for participation of the S3 students whose total usage per student in the second half of the study dropped to under a third of what it was in the first half. This reduced usage may partly explain why there is no statistically significant effect of the Reading Tutor for S3 children in experiment 2. Nineteen children who were absent for testing either at the beginning or end of each experiment are not included in this summary.
Table VIII - Tutor usage in each experiment School Experiment 1 Experiment 2 Number of days of tutor S1 29.6 (12.0) 22.2 (9.4) use per student S2 36.9 (2.4) 24.0 (11.7) S3 30.7 (2.8) 12.6 (3.9) Total time spent reading S1 12.4 (5.4) 11.9 (5.2) with tutor per student S2 18.6 (3.1) 11.5 (6.7) (hours) S3 17.3 (2.7) 6.8 (2.5) Average daily time S1 22.9 (7.1) 32.6 (5.3) spent reading with tutor S2 30.1 (3.7) 26.6 (7.0) per student (hours) S3 33.9 (4.6) 31.9 (5.3)

B. Methodology and Implementation The selection of the school for the pilot test was done after meeting with the headmasters and teachers at both schools and assessing the state of the computer labs and the potential for a successful study. Subsequently, the pilot study consisted of conducting interviews with teachers at the selected school, providing training for the teachers, and making qualitative observations of teachers and students as they used the Reading Tutor. We also explored with the teachers possible details of a longer controlled study. C. Results and Observations Computing infrastructure: Both schools had a computer room with about 20 computers, most of which were 266 Hz Pentium II machines with 64-128MB of RAM and 10GB hard drives. Some teachers from each school had taken a computer skills training course when the computers were initially donated, but had not had the opportunity to reinforce their skills through additional training or guided practice. The schools did not have internet access and the computers were being used primarily for tasks such as typing exam questions, rather than as educational tools for students or for accessing online teaching resources. Given the large class sizes—which ranged from 50 to over 100 students—and strict timetable, the teachers also faced challenges in feasibly using the one lab to teach their students from grades one to nine about computers. Some computers at both schools were not functioning due to broken keyboards, mice, and power strips; some computers were not being protected from dirt during the dusty winter months; power outages were a daily occurrence in Mongu; and for one school, even maintaining electricity for the computer lab was a challenge due to limited financial resources. For the pilot study, we selected the school with better maintained equipment and a higher likelihood of maintaining communication by telephone and email since the school had a telephone and the headmaster had a working email address that he accessed weekly. Training Teachers: We trained three teachers to use the Reading Tutor. The hour-long session covered the educational features of the tutor as well as administrative tasks such as managing users. After basic instruction and some time to practice on their own, the teachers were able to guide students in using the tutor. They would often give feedback to the children as a complement to the tutor when the students had

V. ZAMBIA PILOT STUDY The work in Zambia complements the prior studies in Ghana by investigating an in-school usage model and testing the tutor in a different English-speaking African country. ProjectEDUCATE, a non-profit organization supporting some schools in Zambia, introduced us to two under-resourced public schools in Mongu, the capital of Zambia’s Western Province. The schools had received donated computers and were enthusiastic about the possibility of testing the Reading Tutor. However, the Mongu District Education Board Secretary’s office directed us to select only one school for the pilot test. Three of the authors conducted the study in Zambia. A. Goal The goals of the Zambian pilot study were to answer the following questions: 1. Computing infrastructure: What is the state of the school’s computing lab, how is it currently used, and can it feasibly be used for sessions with a computer-based reading tutor?


difficulty reading stories. For example, they would instruct the children to click for help when they needed it. The teachers would sometimes correct a mispronounced word that the tutor did not detect. We think this involvement of teachers in the early stages as the child gets used to the tutor could be an important part of longer-term use of the Reading Tutor in this school setting. A similar role was played by project staff in the Ghana studies. The teachers especially appreciated the ability to track their students’ performance using the tutor, including the number of new words the children had read and the number of words they read per minute. Children’s response: We observed one group of eleven children in grades two through four as they read one or two stories from the tutor. The students were selected by the teachers and had varying levels of reading ability, English comprehension, and speaking fluency. As in Ghana, all but one student were completely new to computers. We introduced them as a group to the basic components of the computer and then gave them the opportunity to demonstrate use of the mouse and keyboard to each other. We provided verbal instruction to students individually as they began reading stories from the Reading Tutor. Just as in Ghana, the speech recognition capability appeared to work adequately with the Zambian students’ and teachers’ accents. Some Grade 2 students seemed to have difficulty understanding English. This is because English is introduced as a language of instruction only in Grade 2, as a part of the Zambian Ministry of Education’s initiative to encourage basic literacy by teaching in a familiar language in the first year of school [21]. The students enjoyed using the Reading Tutor; when we returned to the school on another day to meet with the teachers, the group of students was using the word building exercises in the Reading Tutor on their own time.

Figure 3 –Teachers and a child test the Reading Tutor in Mongu

Test instruments and Reading Tutor content: We discussed with the teachers other reading assessment options for a longer term study, in addition to the fluency test and Test of Written Spelling. We learned that through the Zambia Primary Reading Programme (PRP), the students’ literacy levels are measured by reading standardized story books aloud to their teachers. Each color-coded book is associated with a given reading level. Once the teacher determines that the child can successfully read at a given level, he/she moves the child up to a higher level reading group with a different set of books. The teachers were interested in engaging the students with the computers in a way that would support curricular requirements from the Ministry of Education, and this was also emphasized to us by the Ministry of Education officials. Accordingly, we discussed incorporating the standardized PRP reading material into the tutor, which will also have the advantage of providing additional content choices for students. We saw that many of the existing Reading Tutor stories, such as those related to baseball, zoos, or recycling, had little relevance to children’s lives in Mongu. Feasibility of long term study: The teachers suggested that a random subset of the children in grades 2 through 4 could be selected to participate in a controlled study. They explained that students in these grades have 60 minutes of reading class per day and suggested that the intervention group of students could spend 30 minutes of that time working with the Reading Tutor, with teacher supervision, while the control group of students would remain in class. We anticipate several challenges in conducting this project as a remote field study. Although we have email and telephone contact information for the key collaborators at the school, we expect communication to be difficult given their limited internet access and the frequent disruptions in telephone communications due to daily power cuts. Success would depend on the teachers taking ownership of the project as a result of their enthusiasm. We hope that communication with local contacts will help address some of the expected challenges. For example, we plan to involve the Mongu District Education Board Secretary’s office in evaluating the project’s progress at the school. We also engaged a local ProjectEDUCATE technical contact who joined us at our meetings with the schools, received training from a member of our team, and assisted us with installation and use of the Reading Tutor software. We hope that he will stay in touch with us remotely, help incorporate new stories into the tutor, check in periodically with the school, and be available to the teachers to assist with any technical challenges. VI. LESSONS LEARNED From the experiences in Ghana and Zambia, we can glean many useful lessons for our future work as well as for others implementing similar studies. These can be broadly categorized into lessons about building relationships and support, lessons about running the study, and lessons about the viability of automated tutoring.

Figure 4 – Students in Mongu using word-building exercises as a group


A. Building Relationships An important requirement for running these studies was the process of developing partnerships at each stage in the project. Significant time must be devoted to building relationships and developing a shared vision. Phone conferences can be a useful tool, but some face-to-face meetings are essential, particularly in the early stages of project planning. A staged implementation, as in a pilot study followed by a controlled study, helps distill key questions to ask, refine design and implementation decisions, and identify potential problems. B. Logistics We learned that it was important for someone involved in the local day-to-day running of the project to have significant decision-making power and to feel ownership of the project. This individual must have the ability, for example, to replace equipment or interface with representatives of the school administration. Otherwise, problems can easily stall progress. It is also essential to have access to some local technical expertise to troubleshoot and repair problems—building this local support base during the pilot studies is crucial. For the controlled study in Ghana, we found that the remote collaboration required regular, sometimes daily, communication among the project partners; email, instantmessaging and voice-over-IP were a cost effective and feasible means of achieving this. Not surprisingly, we learned that a process that requires a significant change in behavior and extra work on the part of parents, such as having the children come to school or participate in a study during a vacation, is hard to sustain and should be avoided if possible. However, it was also clear that unexpected situations are bound to arise for any study that involves cross-continent collaboration and so flexibility and the ability to adjust the study design if necessary is essential. For example, in the controlled study in Ghana, the second half of the study had to be held during the school vacation although this was not the original plan. In running the Ghana studies, transportation of the children was the greatest expense and so although scheduled out-ofschool use in an ICT center was the model chosen for the controlled study, it is not a viable long-term usage model. However, as we learned in Zambia, there are many challenges to be addressed for an in-school usage model for public underresourced schools, even in a school that already has a computer lab. These issues include the limited availability of computers, inadequate experience on the part of the school in maintaining computer labs, scheduling challenges with respect to the school timetable, and also the need to work within the constraints of the existing literacy curriculum. C. Viability of Automated Tutoring A key lesson was that even without prior computer experience, the participating children were quickly able to acquire the skills needed to use the tutor. In general, they were excited about using the computer and about the interactive features of the Reading Tutor. Assistance from the project staff (in Ghana) or from the teachers (in Zambia) helped those children who were initially nervous about computer use or

about reading in English. In both locations, the children’s natural curiosity overcame any initial apprehension of the unfamiliar technology. In observing the children using the Reading Tutor, we noticed that children with a basic foundation in English and not much prior experience with computers, such as the S2 children in Ghana, were easily engaged with the tutor. It was clear that insufficient familiarity with the English language was a challenge for some, particularly the S3 children in Ghana and some of the Grade 2 students in Zambia. For children who did not yet understand English well enough to benefit from an automated tutor that uses only English, a tool that bridges between the local language and English, e.g. by giving prompts or explaining words in the local language, might have been better. Finally, we noticed that the S1 children who were already fluent readers and experienced with computers seemed to get bored and distracted easily when using the Reading Tutor. This might have been because the Reading Tutor displays the story being read one sentence at a time, and there would sometimes be a short delay in loading the next sentence. For a fluent reader, even this short delay was noticeable and sometimes frustrating. There were also instances that indicated that some children got bored of repeated activities and enjoyed variety. For example, many children enjoyed the word-building exercises in the Reading Tutor but would sometimes complain that there were too many of them. Although the Reading Tutor is designed for use by a single user at a time, we noticed that in both pilot studies children would gather around a single computer and try to help each other. This also happened to a lesser extent during the controlled study in Ghana and has been observed in other technology interventions in developing communities [15]. It suggests that we should investigate multi-user scenarios. Finally, there are several usability improvements that can be made to the Reading Tutor to reduce the required technical support, particularly in a developing community setting. These features would apply to any PC-based intervention. • Installation must be easy and straightforward. • Customization must be easy (both for adding local content and to control for the installation footprint for machines without much hard disk capacity). • An easy administrative interface is needed for controlling options such as level of logging, again to deal with limited-capacity machines. VII. CONCLUSIONS AND FUTURE WORK This paper presents our initial experiments in Africa with an automated reading tutor to improve child literacy. It investigates the viability and effectiveness of a computerbased reading tutor in improving the reading ability of children, particularly those attending under-resourced public schools. Although literacy and education for all are at the top of the global development agenda, not much work has been done regarding the role of technology in this process. This


work is a useful contribution both because it demonstrates that there is promise for the effectiveness of the approach and because of the practical lessons regarding the implementation of the study. We have presented an initial proof-of-concept investigation. Many additional questions would need to be explored to understand the potential for large-scale application of these technologies. Next steps for our project include trying to set up a year long in-school study in Zambia. The purpose of the study would be to test the feasibility of an in-school usage model in a developing community setting. It would measure the children’s reading gains over an entire school year of using the Reading Tutor and would compare these gains to those obtained with a non-technological approach to guided reading practice. An important goal of the year-long study will be to learn about sustainability and the feasibility of incorporating teachers closely into this work. Additionally, we will continue to develop an understanding of what features would be required of a literacy-enhancing tool developed specifically for use by children in developing communities. What features would be needed, taking into consideration limited resources and equipment, as well as current instructional practices? What tutorial methods are most useful? Could a lighter weight tool be designed for alternative platforms, such as mobile phones, as has been done with the educational games for English-as-a-second language instruction in the MILLEE project [7]? There is significant scope for further research on the role of computing technology in improving child literacy in developing communities. ACKNOWLEDGMENT We are grateful to Steve Fienberg for his advice on the statistical analysis as part of the Statistical Practice course at Carnegie Mellon University. We express our appreciation to the Ghana-India Kofi Annan Centre for Excellence in ICT (AITI-KACE), Accra, Ghana, for their role in the controlled study, particularly to Dorothy Gordon for her support, Patricia Nyahe for her tireless management of the study, and the UNESCO Ghana Office for sponsoring their work. We also thank Leslie Casely-Hayford of Associates for Change, Ghana, for her advice and contributions to the pilot and controlled studies in Accra. We acknowledge the valuable contributions of Cybercity Internet Café and ProjectEDUCATE to the pilot studies in Ghana and Zambia respectfully. Finally, we are indebted to all the participating children and schools, and to all others who contributed to the project in various ways. REFERENCES
[1] M. J. Adams, "The promise of automatic speech recognition for fostering literacy growth in children and adults,” in M.C. McKenna, L.D. Labbo, R. D. Kieffer, & D. Reinking (Eds.), International Handbook of Literacy and Technology, Volume 2. Mahwah, NJ: Lawrence Erlbaum Associates, 2006. R. L. Allington and A. McGill-Franzen, “The Impact of Summer Setback on the Reading Achievement Gap,” in Phi Delta Kappan 85(1):68-75, September 2003.

[3] [4] [5] [6] [7]

[8] [9]

[10] [11]





[16] [17] [18]


[20] [21]



J. Cohen, Statistical power analysis for the behavioral sciences (2nd ed.), Hillsdale, NJ: Lawrence Earlbaum Associates, 1988. J. Cohen, “A power primer,” Psychological Bulletin, 112, 155-159, 1992. S. L. Deno, “Curriculum-based measurement: The emerging alternative,” in Exceptional Children, 52(3):219–232, 1985. J. Hasbrouck and G. A. Tindal, “Oral reading fluency norms: A valuable assessment tool for reading teachers,” The Reading Teacher, 59(7), 636644, 2006. M. Kam, A. Agarwal, A. Kumar, S. Lal, A. Mathur, A. Tewari, and J. Canny, “Designing E-Learning Games for Rural Children in India: A Format for Balancing Learning with Fun,” Proceedings of ACM Conference on Designing Interactive Systems (DIS '08), Cape Town, South Africa, February 25-27, 2008. S. C. Larsen, D. D. Hammill, and L. C. Moats, Test of Written Spelling, Pro-Ed, Austin, Texas, 1999. M. Lipson and K. Wixson, “Evaluation of the BTL and ASTEP Programs in the Northern, Eastern, and Volta Regions of Ghana,” Report prepared by the International Reading Association for The Education Office, USAID/Ghana, August 2004. Available online at http://www.reading.org/resources/issues/reports/ghana.html R. G. Miller, Simultaneous statistical inference, 2nd ed., Springer Verlag, pages 6-8, 1981. J. Mostow and G. Aist, “Evaluating tutors that listen: An overview of Project LISTEN,” in K. D. Forbus & P.J. Feltovich (Eds), Smart machines in education (pp. 169-234), Cambridge, MA: AAAI Press/The MIT Press, 2001. J. Mostow, G. Aist, P. Burkhead, A. Corbett, A. Cuneo, S. Eitelman, C. Huang, B. Junker, M. B. Sklar, and B. Tobin, “Evaluation of an automated Reading Tutor that listens: Comparison to human tutoring and classroom instruction,” Journal of Educational Computing Research, 29(1), 61-117, 2003 J. Mostow, G. Aist, C. Huang, B. Junker, R. Kennedy, H. Lan, D. Latimer, R. O’Connor, R. Tassone, B. Tobin, and A. Wierman, “4Month evaluation of a learner-controlled Reading Tutor that listens,” In V. M. Holland & F. P. Fisher (Eds.), The Path of Speech Technologies in Computer Assisted Language Learning: From Research Toward Practice (pp. 201-219), New York: Routledge, 2008. National Institute of Child Health and Human Development, Report of the National Reading Panel. Teaching children to read: An evidencebased assessment of the scientific research literature on reading and its implications for reading instruction (NIH Publication No. 00-4769), Washington, DC: U.S. Government Printing Office, 2000. U. S. Pawar, J. Pal, K. Toyama, “Multiple Mice for Computers in Education in Developing Countries,” International Conference on Information and Communication Technologies and Development, ICTD 2006, pp.64-71, May 2006. R. Poulsen, P. Hastings and D. Allbritton, “Tutoring Bilingual Students with an Automated Reading Tutor That Listens,” Journal of Educational Computing Research, 36(2), 191-221, 2007. Project LISTEN Videos, available at http://www.cs.cmu.edu/~listen K. Reeder, M. Early, M. Kendrick, J. Shapiro, and J. Wakefield, “The Role of L1 in Young Multilingual Readers’ Success With a ComputerBased Reading Tutor,” Talk at the Fifth International Symposium on Bilingualism, Barcelona, Spain, April 2005. M. R. Shinn, N. Knutson, R. H. Good, W. D. Tilly, and V. L. Collins, “Curriculum-based measurement of oral reading fluency: A confirmatory analysis of its relation to reading,” School Psychology Review, 21:459-479, 1992. K. E. Stanovich, Progress in Understanding Reading: Scientific Foundations and New Frontiers, New York: Guilford Press, 2000. UNESCO, “Education for All Global Monitoring Report 2005 – Education for All: The Quality Imperative,” United Nations Educational, Cultural and Scientific Organization (UNESCO) Publishing, 2005. Available online at http://portal.unesco.org/education/en/ev.phpURL_ID=35939&URL_DO=DO_TOPIC&URL_SECTION=201.html UNESCO, “The Plurality of Literacy and its Implications for Policies and Programmes.” UNESCO Education Sector Position Paper. United Nations Educational, Scientific and Cultural Organization, Paris, 2004. Available online at http://unesdoc.unesco.org/images/0013/001362/136246e.pdf


Improving Literacy in Rural India: Cellphone Games in an After-School Program
Matthew Kam, Anuj Kumar, Shirley Jain, Akhil Mathur, and John Canny
language of all professions and higher education, but also important for mid-level service jobs: retail, clerical, teaching, law enforcement, etc. that are the most common steps above menial labor. The value of English is widely recognized by ordinary Indians [2], and it is in fact the poorest citizens who are lobbying most strongly to expand English teaching. English is thus the language of power in India associated with the middle and upper classes [3][4]. In other developing regions, it is another language such as Spanish, Mandarin, or French which is not native to most of the population. We believe that many of our lessons will apply to other languages although our focus is on English as a Second Language (ESL). But the public school systems in developing regions face insurmountable difficulties. In India, for example, we were consistently unable to converse in English with those teachers responsible for teaching English in poor schools, where the overwhelming majority of children in the country struggle to learn. More important, public schooling is out of the reach of large numbers of children in rural areas and the urban slums who cannot attend school regularly, due to their need to work for the family in the agricultural fields or households [5]. At the same time, cellphones are increasingly adopted in the developing world, and an increasing fraction of these phones feature multimedia capabilities for gaming and photos. These devices are a promising vehicle for out-of-school learning to complement formal schooling. In particular, we believe that ESL learning games on cellphones present an opportunity to dramatically expand the reach of English learning, by making it possible to acquire ESL in out-of-school settings that can be more convenient than school. Games can make learning more engaging while incorporating good educational principles [6]. More important, a large-scale evaluation with urban slums children in India has shown significant learning benefits from games that target mathematics [7]. We believe that similar outcomes can be replicated with e-learning games that target literacy. The challenge in evaluating any language learning project, however, is that language acquisition is a long-term process on the learner’s part. Worse, with a novel technology solution that has yet to be institutionalized, there were tremendous logistical obstacles in running a pilot study over a non-trivial duration. After 3 years, in which we commenced with needs assessments and feasibility studies, followed by subsequent rounds of field testing interleaved with numerous iterations on our technology designs, we have established the necessary relationships with local partners for such an evaluation. This

Abstract—Literacy is one of the great challenges in the developing world. But universal education is an unattainable dream for those children who lack access to quality educational resources such as well-prepared teachers and schools. Worse, many of them do not attend school regularly due to their need to work for the family in the agricultural fields or households. This work commitment puts formal education far out of their reach. On the other hand, educational games on cellphones hold the promise of making learning more accessible and enjoyable. In our project’s 4th year, we reached a stage where we could implement a semester-long pilot on cellphone-based learning. The pilot study took the form of an after-school program in a village in India. This paper reports on this summative learning assessment. While we found learning benefits across the board, it seemed that more of the gains accrued to those children who were better equipped to take advantage of this opportunity. We conclude with future directions for designing educational games that target less well-prepared children in developing regions. Index Terms—cellphone, English as a Second Language (ESL), literacy, mobile game, pilot study

ITERACY is one of the great challenges in developing regions. Despite huge improvements in recent decades, literacy levels in many poor countries remain shockingly low. Even more challenging is the tension between regional and global “power” languages, that economic opportunities are often closed to those literate only in a regional language. For instance, India is a country with 22 regional and 2 national languages, i.e. Hindi and English. But English, together with computer skills, are the two most requested skills in surveys of poor parents [1]. English is a great economic enabler. It is the
Manuscript received September 22, 2008. This work was supported in part by the U.S. National Science Foundation under Grant 0326582, a Qualcomm BREW Wireless Reach award, and sponsorship-in-kind from Sony Creative Software. Matthew Kam is an Assistant Professor in Carnegie Mellon University’s Human-Computer Interaction Institute (phone: +1 412-268-9805; fax: +1 412268-1266; email: mattkam@cs.cmu.edu). He was with the Berkeley Institute of Design and the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley when this work was carried out. John Canny is the founding director of BID and holds the Paul and Stacy Jacobs Distinguished Professorship of Engineering at UC Berkeley. Shirley Jain participated in the pilot study as a curriculum developer and local supervisor. Anuj Kumar and Akhil Mathur are with the Dhirubhai Ambani Institute of Information and Communication Technology, Gujarat, India.




paper describes the results from a semester-long pilot study – the longest so far in this project – which took place during the project’s 4th year. The study involved 27 rural children who participated in an after-school program that we implemented in their village.

conducting a learning assessment for a novel educational technology over a substantial timeframe when it is not yet integrated into the everyday operations of a formal entity.

III. OVERVIEW OF PILOT STUDY II. RELATED WORK Most work on technology-assisted language learning in the developing world does not explore the convenience that the cellphone’s mobility offers. Banerjee et al. [7] report a largescale evaluation with mathematics learning games on desktop computers, carried out over 2 years with urban slums children in India. Mitra et al. [8] describe a study in India with slums children over 5 months, which involved a “hole-in-the-wall” public computer installed with speech-to-text software. Dias et al. describe a computer-based tutor software for improving reading in Ghana [9]. Kothari’s karaoke-like approach [10] targets native language – not second language – literacy using television. With the cellphone’s increasing ubiquity in Africa, Brown [11] argues that it is timely to envision a future where the cellphone plays a pivotal role in education in Africa. Kam et al. [12] describe how a set of ESL-learning cellphone games that targets children in rural India have undergone numerous iterations, based on successive, short-term formative studies. With the exception of Kam et al., the only education-related projects we know of in the developing world that leverage the cellphone are Islam et al. [13] in Bangladesh, and Librero et al. [14] in Mongolia and the Philippines. Both projects rely on Short Messaging Service and target university students, unlike our approach. To date, Horowitz et al. [15] is the only study we know of that examines the cellphone for promoting literacy. However, this study took place in an industrialized country (USA), even though participants included households below the poverty line. In the study, Sesame Street videos that target the English alphabet were streamed to preschool children over cellphones throughout an 8-week period. Our paper therefore contributes to the literature as the first learning assessment on cellphonebased language instruction in developing regions. He et al. [16] describe a 2-year randomized evaluation of a LeapPad-like device that supports custom software modules for English learning. This interactive system involves a paper book attached to a stylus and supports audio output. It lacks a visual display, unlike a cellphone, but overlaps with our goal of making literacy learning more accessible in the developing world using portable devices. Among the learning technologies for developing countries, one of the most novel devices is the multiple-mice computer described in Pawar et al. [17]. It was intended for collocated learning by a group of children around each computer, with a mouse input device for each child. It has since been extended to distance learning in Moraveji et al. [18]. The evaluations in both papers were short-term, and underscored the difficulty in The pilot study was carried out in collaboration with a nongovernment organization in North India under the terms of a Memorandum of Understanding. The study took the form of an after-school program, which we held during the afternoons at a private village school affiliated with this NGO. However, our goal was to investigate learning impacts that ESL learning games on cellphones have on lower-income rural children. As such, students who were already enrolled in this school were ineligible to participate in the study. Instead, we invited those parents who could not afford the fees for this private school – and hence sent their children to less expensive schools in the same area – to give consent for their children to participate. In the after-school program, we ran three sessions per week, on average. Each session lasted two hours in the afternoon. Children from neighboring villages attended the after-school sessions after finishing their regular classes in the morning. In the after-school sessions, we loaned cellphones preloaded with ESL learning games to participants. The after-school program took place from late December 2007 to early April 2008, and spanned sessions on 38 days in total.

IV. DATA COLLECTION As our preparation for this pilot study, we made two trips to India, i.e. once in the summer of 2007 to familiarize ourselves with the pilot location and end-user community, and a second time in December 2007 to kick-off the actual pilot. 4 local staff members were hired to run the after-school sessions on an everyday basis. 3 of them were engineering undergraduates in their last semester, while the last member had graduated a few years ago. We spent two weeks training them to run the after-school sessions and perform data collection, and continued to coordinate with them regularly via conference calls and emails after we left India. We interviewed participants on their demographics such as their ages and the grades they were currently enrolled in in school. During the interviews, we also asked other questions, such as the number of cellphones that their households owned, what they currently and/or had previously used cellphones for, their television watching habits and frequency, as well as their parents’ occupations. The questions on media and technology exposure were included because these variables were expected to impact participant ability to learn using cellphone games. To ensure that each participant has the basic numeracy and ESL literacy to benefit from cellphone-based learning in the program, participants were required to pass a qualifying test, i.e. obtain at least 50% of the total score. The test required them to complete one-word blanks using English words about themselves, e.g. name, age, school, grade, etc. They were also


asked to fill in the missing letters in the alphabetic sequence, write numbers in the Arabic notation, match words with their pictures, spell the words for everyday objects, and describe a picture of a market scene with short sentences. The qualifying test was designed such that an average child in India with no learning disabilities who has finished 1st grade in a reputable urban school should obtain a perfect score on it. By using the qualifying test as a screener, we ensured that participants were numerate. This was important because we had previously found it difficult to teach children to use the cellphone keypad’s to play e-learning games when they were not familiar with the numbers from 0 to 9 in Arabic notation. Similarly, by ensuring that participants were familiar with the English alphabet, we could target a more advanced syllabus that went beyond the alphabet. We made this decision since Horowitz et al. [15] had already investigated the efficacy of cellphone-based learning for the English letters in the context of preschool children in the USA. Since success in acquiring a second language is correlated with literacy in one’s native language, we administered a test which evaluated the ability of the participants to read in Hindi. Every child was given a short passage that described a diet for promoting dental health. Each child was then asked to read the passage aloud so that we could observe his or her fluency and accuracy. These sessions were videotaped. After that, every participant was asked to write answers to written questions that tested his or her comprehension of the passage, in Hindi. We had designed this test such that an average child who has finished 3rd grade in a reputable urban school in India should obtain a perfect score on it. Our primary method of assessment was to administer preand post-tests which evaluated participants on their ability to spell the common nouns that the curriculum for the pilot study targeted. Although the curriculum targeted other competencies such as listening comprehension and the recognition of written words, our assessment emphasized spelling, which as a recall task was cognitively more difficult than recognition tasks. We maintained attendance records for the participants for every session. We also videotaped each session so as to have contextual data that could potentially account for their test performances. The video recordings captured the classroom proceedings, and individual participants’ interactions with the games. The latter recordings captured participants’ levels of engagement with the games as shown in their facial and body expressions. The pilot staff member who was responsible for videotaping the sessions tried to ensure that every participant was videotaped playing at least one level in the curriculum per day. The recordings were later transcribed and translated from Hindi to English. Finally, for every session, we asked pilot personnel to write a report which summarized what happened in that session, as well as how well each participant interacted with the games. The latter not only covered usability and learning obstacles, but also included pilot staff’s observations on the attitude and persistence that each child demonstrated towards learning.

V. PARTICIPANTS Owing to the strong relationships that our NGO partner had built with the local community over more than a decade, we were able to generate a high level of support among parents in this community. In total, we obtained consent for 47 children to participate in the study. However, we needed to turn 16 of them away; 15 children did not pass the qualifying test while the 16th was attending private tuition for English. The latter represented a confounding variable. Of the 31 children whom we started the pilot with, 4 of them left the program mid-way. Reasons for attrition include time conflicts with private tuition (2 children) and disinterest in attending the sessions (another 2 children). From post-deployment interviews, we understand the latter was due to caste tensions between those 2 children, who belonged to the lower castes, and some upper-caste children in the program. A. Demographics The 27 children who participated in the study until it ended were aged 7 to 14 (mean = 11½ years) and belonged to grades 2 to 9 (mean = 6th grade). There were 11 boys and 16 girls. 5 children came from the upper castes while others belonged to the lower castes. The gender and caste breakdown seemed to mirror the demographics in the community. Every participant attended between 8 and 29 sessions (mean = 20) in the afterschool program, broken down according to the following three functions: • Cellphone training: 0 to 5 sessions (mean = 4) where we taught participants how to use the cellphones, perform alphanumeric input and play mobile games, • ESL learning: 4 to 17 sessions (mean = 10) in which participants played ESL learning games on the cellphones, and • Assessment: 4 to 7 sessions (mean = 6) for administrative tasks and data collection, e.g. demographics interviews and various tests. In India, traditionally, only the upper castes owned land. As such, the upper castes earn their livelihood on the land or run small businesses, while lower castes graze their goats, work as daily-wage laborers or perform menial jobs in the homes of the upper castes. Land-owning and non-land-owning families told us that they earned up to 100,000 (US$2,500) and 50,000 rupees (US$1,250) respectively per year. B. Hindi and English Baseline 26 of the 27 participants were enrolled in the same school, where Hindi is the medium of instruction. The last participant was a school dropout. Assuming regular school attendance, the typical participant would have taken classes on Hindi and English for 5½ and 3½ years respectively prior to the study. We devised a grading rubric to evaluate each participant on the Hindi literacy test and qualifying test. On the former test, participants scored 7.9 out of 18 on average (σ = 4.5, n = 19). 2 participants turned in blank answer sheets. We observed the following problems in the submissions:


• Wrong answers due to poor comprehension of the questions (5% of the test-takers) or passage (53%), or responses that simply repeated the questions (32%) • Spelling errors (16%) • Grammatical errors, i.e. using the incorrect form of the verb for the subject’s gender (21%), or the incorrect form of the noun for the subject’s singularity vs. plurality (5%) • Inability to phrase responses in complete sentences (5%)
TABLE I BREAKDOWN OF PARTICIPANT PERFORMANCE ON QUALIFYING TEST Poor Fair Good 13% left blanks 32% filled in blanks 55% filled in blanks About empty or filled them with at least 1 with correct myself
in Hindi misspelling


Word recognition Spelling Picture description

5% filled in less than 7 blanks in alphabetic sequence correctly 14% matched up to 2 out of 6 words with correct pictures 41% spelt up to 2 out of 6 words correctly 18% made no attempt to answer this section

13% filled in ~11 blanks in alphabetic sequence correctly 14% matched 4 out of 6 words with correct pictures 18% spelt about 3 out of 6 words correctly 50% wrote answers as individual words, not full sentences

spellings for most questions 82% filled in at least 12 out of 13 blanks in alphabetic sequence correctly 72% matched at least 5 out of 6 words with correct pictures 41% spelt at least 4 out of 6 words correctly 32% wrote intelligible answers in sentences

participants came from families that owned cellphones with a color screen, as opposed to monochrome display. Two of the above cellphones – both of which belonged to 2 of the above 3 boys – contained built-in cameras. Nonetheless, cellphone ownership and access were separate issues. Among the 25 children whose families owned at least one phone, 6 of them – 5 girls aged 7-11 and a boy aged 12 – were prohibited by their parents from using the phone, either entirely or most of the time. In general, in poorer families, it seemed that children were allowed to receive (free) incoming calls, but not play mobile games lest they drop the devices. On the other hand, in wealthier families, children were allowed to play on the phones. As such, although most participants were familiar with cellphones, it appeared that rural parents were more willing to entrust these relatively costly devices to their sons (vs. daughters). In total, 15 of the 27 participants (56%) reported that they had played cellphone games before.

VI. CURRICULUM AND GAME DESIGN One of the major challenges with carrying out a pilot study over a non-trivial timeframe was that we needed to develop sufficient digital content that could last throughout its entire duration. We ensured that our syllabus was aligned with local ESL learning needs in India by recruiting a local ESL teacher as our curriculum developer. She had a decade’s experience as an ESL teacher at a prestigious urban school, located in the same geographic region as the after-school program. A. Curriculum Design Given the above attendance rate, the ESL curriculum for the pilot was designed to be comparable to the amount of material that a qualified teacher could reasonably cover in 18 hours with rural children in a classroom. The syllabus was situated within the classroom theme, which participants could readily relate to. Concretely, the syllabus included: • Common nouns that are found in the typical classroom, e.g. chair, table, door. • Verbs that can be performed with the above nouns, e.g. sit, write, open, close. • Sentence structures for constructing sentences out of the above nouns and verbs, e.g. “This is a __.” • Sentence structures for phrasing question-and-answer sequences with the above nouns and verbs, e.g. “What is this?”, “Where is the __?” The curriculum design took participant performance on the qualifying test into account. The curriculum was also based on our attempts to converse informally with participants, during which we learned they did not comprehend simple questions about themselves, did not know the English words for objects around them (e.g. in the classroom), and made grammatical errors. The curriculum therefore targeted the above syllabus in terms of listening comprehension, word recognition (of the written word), sentence construction and spelling.

On the qualifying test, on average, participants scored 44.0 out of 50 (σ = 5.5, n = 22). Our grading rubric indicated how participants should be classified as “poor,” “fair” and “good” on every section of the test. Table I gives the breakdown of how test-takers were distributed across categories for selected sections, and descriptions of the categories. In summary, the average participant had a good knowledge of the alphabet and a fair vocabulary of written words that she could read. On the other hand, she was weak in recalling and spelling everyday nouns, and even weaker in constructing complete sentences with these words. Despite the wide range in the ages of the participants, it appeared from their performance on the above tests that the variation in their English proficiency was much narrower. More specifically, we estimated that the average participant was comparable to an urban child in India who had taken between 1 and 2 years of English classes. Notably, only 10 children (45%) could spell their names correctly in English on the qualifying test papers. We had a chance later to interview the teacher who taught them English in their school. She revealed that her pedagogical approach revolved around having students copy sentences from English textbooks into their notebooks. She felt that it was not worth putting in more effort to teach English since she believed she was underpaid. C. Technology Baseline Among the 27 participants, 25 of them came from families who owned at least one cellphone; 5 participants belonged to families that owned 2 phones each while 2 participants came from families which owned 3 phones each. The cellphone was usually used by the eldest male member in every family, and in fact, 3 boys aged 13-14 possessed their own cellphones. 8


B. Game Design We designed a set of ESL learning games for the cellphone platform that targeted the above curriculum, and piloted them in the after-school program. Our designs drew on 3 resources, namely: 1) recurring patterns in state-of-the-art commercial software applications for language learning, which represented best practices that we reused to avoid reinventing the wheel, 2) traditional village games, which more closely matched the expectations and understandings that rural children have about games, in comparison to contemporary videogames that were largely Westernized, and 3) lessons from several previous rounds of field-testing and iterations with rural children elsewhere in India [12]. In this subsection, we walk the reader through a subset of the screen designs.

Fig. 3 and 4. Figures 3 and 4 introduce additional phrases that the nouns and verbs in the syllabus can be used in. Figure 3 teaches a phrase that associates the verb “sit” with the noun “chair.” Figure 4 shows how to ask questions using the “Where” keyword. Abstract phrases and function words such as “where” are difficult to convey graphically. Hence, when they are taught for the first time, the software explains their meanings orally in Hindi.

Fig. 1 and 2. Figure 1 introduces the English vocabulary for common nouns in the classroom. Word-picture association is a technique employed by many successful commercial language learning software. As the boy moves to each object, the software highlights the corresponding word in a different (green) color and says the word aloud. Figure 2 situates these objects in a classroom scene and builds on the earlier screen by demonstrating how to use the nouns in complete sentences. As the boy moves to each object, the software says the “This is a __” phase aloud for the corresponding object.

The games tested players on their comprehension and recall of the words and phrases. For example, the game shown in Figure 5 says the word aloud for one of the objects displayed on screen. The player needs to identify the correct object and push it onto the area that is blinking blue. At the same time, he needs to avoid the balls thrown by the computer-controlled opponent. This game was an adaptation of Giti Phod, which was one of the traditional games that children play in Indian villages. In Giti Phod, players in a team have to arrange some objects (e.g. rocks) into a given configuration (e.g. a heap), while avoiding being hit by a ball thrown by members in the opposing team. In our experience, we have observed that rural children found it more intuitive to understand videogame rules when the designs of these videogames drew on the rules found in the traditional village games that they play everyday.

In earlier field studies, we observed that rural children did not readily associate a game with learning. It seemed that they viewed a game as an activity to be played purely for pleasure, and did not pay attention to the educational content embedded within game activities. On the other hand, when educational content such as English words and phrases were introduced in non-interactive screens separate from interactive game screens, the rural children appeared to grasp more intuitively that the software was trying to teach them those English words and phrases. Users subsequently paid more attention to the latter. Figures 1 to 4 show some screenshots in which we introduced words and phrases – both written and spoken – to the learners.

Fig. 5. A word-picture matching game which is an adaptation of one of the traditional village games that children in rural Indian play everyday.

Given that television has become a pervasive media among all economic classes in India, it only made sense to draw on popular culture in India to make our designs more appealing to children there. One of these sources is Sesame Street, which is a successful television program for young children that has local co-productions around the world – in both industrialized and developing countries. Its producers in India have found some of its localized characters to be popular with children in India, and we incorporated those characters into our designs


for teaching (Figure 4), quizzing (Figure 6) and congratulating (Figure 7) the user.

Fig. 6 and 7. Localized characters from the Indian production of Sesame Street tested the player on his ability to engage in question-and-answer style dialogues, and performed a victory dance for the player upon successful completion of each level in the game.

Fig. 9. The second level of hints in the spelling activity. For every blank, a set of possible letters are displayed to provide the player with some assistance, if he was unable to spell the word correctly after two attempts.

The activity that targets spelling skills is shown in Figure 8. The player is given an image (e.g. blackboard) and is required to spell the word corresponding to it. Some of the letters in the word are displayed, while blanks are shown for the remaining letters. The player moves between blanks with the arrow keys. Once he has filled in all the blanks and submitted his answer, the correct and wrong letters are displayed in green and red respectively. This feedback constitutes the first level of hints that we have designed to help the learner arrive at the correct spelling. If the player spells the word correctly, he proceeds to the next game. Otherwise, all blanks are cleared after a short pause and he is required to spell the same word again.

The curriculum is broken up into a total of 6 levels in the games. On every screen, the player can access a menu through a shortcut button. Among various options, this menu permits him to move to an earlier level in the curriculum to repeat the material, as well as to move to higher levels in the curriculum. The software was designed so as not to require airtime, which was expensive for most rural families. We implemented the games on Adobe’s Flash Lite and Qualcomm’s BREW (Binary Runtime Environment for Wireless) platforms. We piloted the games on Motorola’s Razr V3m cellphone model, which has a fairly large screen. VII. PILOT SESSIONS In those sessions where participants were taught how to use the cellphone, they were shown how to move their sprites with the arrow buttons. They were also taught how to perform alphanumeric text entry, since most of them did not know this. Sprite movement and text input were essential skills for the games we designed. Pilot staff were therefore asked to write some simple, short sentences on the blackboard, and ensure that each participant demonstrated his ability to enter those sentences via text input. Some other sessions focused on administrative tasks, such as the above tests and demographics interviews. We learned that a few participants had difficulty reading a small subset of the English alphabet despite having passed the qualifying test. We spent two sessions coaching them on those less-frequently encountered letters, so that they would be better prepared for the syllabus targeted in the pilot. Next, at least 8 children had seen the localized Sesame Street characters on television, but did not know their names. To help participants better relate to the characters, so that our games would appeal to them even more, we introduced the characters at the start of the semester. We also screened 3 localized episodes on separate occasions. These episodes were chosen such that they were educational but did not target English learning. Each episode lasted ½ hour, and we observed that participants enjoyed the humorous acts performed by the characters.

Fig. 8. The first level of hints in the spelling activity. After the player has tried to spell the word by filling in the blanks with letters, the correct and incorrect letters are shown in green and red respectively.

If the player is unsuccessful in spelling a word correctly after two attempts, the second level of hints (Figure 9) appears to provide him with additional learning support. Based on the blank that the cursor is currently located at, the game displays a set of possible letters for him to narrow down the choice of candidate letters.


Fig. 10. In the after-school sessions, each participant was loaned a cellphone preloaded with English language learning games. Participants were taught how to start the games, and were asked to focus on learning English when playing the games on their own.

Fig. 11. Frequency histogram of participant scores on the pre- and post-tests.

The remaining, and majority of, sessions focused on ESL learning. A 2-hour session was typically structured as follow: after an exchange of greetings, pilot staff took attendance and briefed participants on the learning objectives for that day. If new games were deployed that day, pilot staff explained and demonstrated how to play them to the participants in small groups. Each participant was then handed a cellphone to play the games on her own (Figure 10), and were told to focus on learning the English syllabus that the games covered. Children who were absent on previous sessions received help from pilot personnel in learning how to play those games that they were unfamiliar with. Pilot staff were limited to providing technical support; and were explicitly instructed not to teach English or communicate with participants in English. There was a short break of 10 to 15 minutes in the middle of each session. At the end of each session, pilot staff took back the phones so that they could charge their batteries overnight and download new games onto them. Each participants received a small packet of biscuits after every session. VIII. QUANTITATIVE RESULTS On the pre- and post-tests, test-takers were awarded 1 point for each common noun in the syllabus that was spelt correctly. A. Post-Test Gains The mean pre-test score was 5.2 out of 18 (σ = 3.3, n = 27) while the mean post-test score was 8.4 out of 18 (σ = 5.5, n = 24). Participants exhibited significant post-test gains on a onetailed t-test (p = 0.007). We present the frequency histograms for both scores in Figure 11. They illustrate that the score distribution had shifted toward the higher end of the spectrum after the deployment.

The average post-test gains was 3.4 out of 18 (σ = 3.3, n = 24). The gains exhibited a fairly large variation, and ranged from -2 (two participants exhibited negative gains) to 9 out of 18. We present the frequency histogram for post-test gains in Figure 12.

Fig. 12. Frequency histogram of participant post-test gains.

B. High-Gains vs. Low-Gains Learners We sought to understand how participants’ post-test gains were correlated with their demographics and performance on other tests. We also categorized participants into two groups, namely, “high-gains learners” and “low-gains learners,” based on their post-test gains. A learner whose post-test gains exceeded the mean of 3.4 was categorized as a “high-gains learner,” else he was classified as a “low-gains learner.” In all, 9 participants were classified as high-gains learners while 15 participants were categorized as low-gains learners. 3 of the 27 participants could not be classified since they were absent on the day when the post-test was administered. On a normalized scale, when the 27 participants were taken as one group, the average pre-test score was 29% whereas the average post-test score was 47%. The latter score did not seem high in absolute terms, i.e. on average, a participant could not spell over half of the common nouns targeted in the syllabus by the end of the intervention. However, once the participants had been classified, on a normalized scale, high-gains learners scored 41% (80%) on the pre-test (post-test) whereas lowgains learners scored 19% (27%) on the pre-test (post-test), on average. In other words, high-gains learners not only showed larger post-test gains but also appeared to have a higher mean


pre-test score at the start of the intervention. More important, post-test gains for both high-gains (p < 0.001) and low-gains learners (p = 0.076) were significant, with effect sizes of 0.54 and 2.24 respectively. That is, both categories of participants exhibited learning gains. (But average post-test gains for lowgains learners were only marginally significant, because of the 2 participants who obtained lower scores on the post-test compared to their pre-test.)
TABLE II HIGH-GAINS VS. LOW-GAINS LEARNERS IN TERMS OF DEMOGRAPHICS Age Grade Days Days (Years) Enrolled Spent Spent in School Learning Playing ESL ESL Games Games LowMean 10.7 5th 3.9 10.5 gains 1.8 1.9 1.4 4.2 σ learners Min. 7 2nd 0 5 (n=15) Max. 14 9th 5 16 Highgains learners (n=9) Mean

The proportions in Table III were presented for the sake of completeness. We were unable to perform any statistical tests on these proportions due to the small sample size, which for example did not satisfy the standard binomial requirement. We thus caution the reader against drawing firm conclusions from these statistics. However, when examining individual learners to identify surprising cases, we took the demographic variables in Table III into consideration. The analysis is deferred to the following section.
TABLE IV HIGH-GAINS VS. LOW-GAINS LEARNERS IN TERMS OF TEST SCORES QualifyQualifyHindi PrePosting Test ing Test, Literacy Test Test (out of Spelling Test (out (out 50) Section (out of of 18) of 18) (out of 6 18) words)* LowMean 42.9 1.2 6.3 3.5 4.8 gains 2.9 1.1 4.2 2.3 2.4 σ learners Min. 37 0 0 0 2 (n=15) Max. 46.5 4 14 10 12 Highgains learners (n=9) Mean


12.8 1.2 11 14 Yes
(p = 0.002)

8th 0.9 7th 9th Yes
(p < 0.001)

3.8 0.8 2 5 No
(p = 0.4)

10.2 5.2 4 17 No
(p = 0.4)

Min. Max. Is difference between means significant? Correlation with post-test gains (r)


47.1 1.8 43.5 49 Yes
(p < 0.001)

3.4 1.6 2 6 Yes
(p = 0.001)

12.0 1.7 10.5 14 Yes
(p < 0.001)

7.4 3.1 2 13 Yes
(p = 0.003)

14.4 3.6 6 18 Yes
(p < 0.001)





Min. Max. Is difference significant?

In Tables II and III, we examined how high-gains learners may differ from low-gains learners in terms of demographics. On the whole, the high-gains learners did not appear to differ significantly from low-gains learners in terms of the number of days that they spent on learning how to play the cellphonebased games (p = 0.4) or actually playing the games to learn ESL (p = 0.4). Instead, high-gains learners belonged to higher ages (p = 0.002) and were enrolled in more advanced grades in school (p < 0.001). In fact, post-test gains exhibited high correlation with grade levels that participants were enrolled in school (r = 0.61) and medium correlation with age (r = 0.45).

Correlation with 0.57 0.70 0.45 0.46 0.86 post-test gains (r) *In this column, we present the number of words that participants spelt correctly on the spelling section of the qualifying test, out of a total of 6 words.

TABLE III HIGH-GAINS VS. LOW-GAINS LEARNERS IN TERMS OF DEMOGRAPHICS Sex Caste Media Exposure Attitude* 67% 77% 73% have played 24%, 38% & 38% Low(33%) (23%) games on cellphones were described as gains were belonged prior to pilot; below average, learners females to lower 60% (40%) watched average and (n=15) (males) (upper) less (equal to or more) above average

Table IV compares the high-gains and low-gains learners in terms of test scores. The former outperformed the latter on the Hindi test (p < 0.001). Next, we analyzed the qualifying test results at two levels, namely, the score for the entire test as well as the score on the spelling section. We found that highgains learners outperformed low-gains learners on the entire test (p < 0.001) as well as on the spelling section (p = 0.001). High-gains learners also obtained higher scores on the pre-test (p = 0.003) and post-test (p < 0.001), vis-à-vis low-gains learners. In fact, participants’ post-test gains exhibited a high degree of correlation with their qualifying test scores, for both the entire test (r = 0.57) and spelling section (r = 0.70). On the other hand, post-test gains had a lower correlation with Hindi literacy levels (r = 0.45) and pre-test scores (r = 0.46). IX. QUALITATIVE RESULTS The above quantitative results suggested that current levels of spelling proficiency and grades enrolled in school were the strongest predictors of success in learning how to spell new words through the cellphone-based games which we designed. Higher levels of Hindi literacy and academic preparation were also associated with higher post-test gains. On the other hand, the number of sessions that participants had with the cellphone games – both for learning how to play the ESL learning games and learning ESL through the games

Highgains learners (n=9)

44% (56%) were females (males)

88% (12%) belonged to lower (upper) castes

than 1 hour of TV per day 56% have played games on cellphones prior to pilot; 40% (60%) watched less (equal to or more) than 1 hour of TV per day

learners respectively 29% and 71% were described as below average and above average learners respectively

*The Attitude column is based on the observations that pilot personnel have on the seriousness and aptitude that participants exhibited as learners throughout the pilot. These qualitative comments were subsequently coded into the “below average”, “average” and “above average” learner categories.


– were not associated with post-test achievements. Among the 24 children whom we have post-test gains data on, 5 of them were classified as high-gains learners despite having played the games on only 4 to 7 days (mean = 6 days). Conversely, 7 participants were classified as low-gains learners in spite of having played the e-learning games for 13 to 16 days (mean = 14.7 days). More important, pilot personnel described 6 of these 7 low-gains learners as “hardworking” or “serious” about learning ESL. Similarly, we were curious about how the two students who exhibited negative post-test gains, as well as the school dropout, had interacted with the cellphone-based games. The above quantitative trends raise the following questions: How did some of the high-gains learners played the games such that they benefited despite lower attendance? In the case of some low-gains learners, why did they improve little on the post-test despite spending numerous days with the games and being perceived as diligent? In order to address such questions at the interaction design level, we turned to our video records and daily reports. Our hope was to recommend improvements to the technology designs and/or after-school setting. A. Interaction Patterns with the Technology At first glance, it seemed that participants needed to attend the after-school program for more days. Our video recordings showed that only 3 of the participants reached the last level in the curriculum by the last session in the program. This was a surprise. Given that the curriculum was designed for 18 hours of instruction, we expected an average attendance rate of 10 gameplay sessions to constitute enough time with the games. On examining the video recordings more closely, we saw that at least 8 participants were using the game menu to skip ahead to other levels whenever they were unable to spell the words in the current level correctly after a few attempts. (We note that the menu was not necessarily a negative feature. Among those 8 participants, at least 2 of them used the menu to skip those words that they already knew how to spell.) We needed to understand why learners gave up on retrying the spelling activity for difficult words despite the hints in the spelling activity. On the whole, we observed 4 different levels of behavior associated with the spelling activity in the videos: 1) When students encountered a word that they could spell, they pressed the keypad buttons quickly and with ease to fill in the blanks for the missing letters. 2) When students saw a word that they did not know how to spell, some of them learned to spell it correctly with the help of the first level of hints. 3) Some of those students who failed to learn how to spell a word with the first level of hint eventually learned how to spell it correctly with the help of the second level of hints. 4) Other students never succeeded in learning how to spell certain words despite both levels of hints. In general, we observed that high-gains learners succeeded in learning how to spell words after having seen their written forms displayed on earlier screens (i.e. such learners were able to spell those words correctly – without requiring any hints –

on their first attempt in the spelling activity), or with only the first level of hints. It seemed that they did not require much scaffolding support from the software. In fact, from the video recordings of 9 high-gains learners, we saw that 5 (56%) and 1 (11%) of them depended on the first and second levels of hints respectively. In contrast, 12 (80%) and 8 (53%) out of the 15 low-gains learners who were videotaped relied on the first and second levels of hints respectively. It seemed that the low-gains learners, as compared to the high-gains learners, were less able to rectify their errors in filling in the blanks for the missing letters through the first level of hints, and required the second level of hints to attain the correct spellings. Worse, the inability on the part of the low-gains learners to spell correctly with help from only the first level of hints made some of them visibly unhappy or bored when the second level of hints appeared. The reason for this distress was unclear. The learner could be frustrated that he was spending too much time to learn how to spell the word. Alternatively, on seeing the second level of hints show up, he could be demoralized that he had just been relegated to the ranks of the most inferior learners and needed the second level as a “crutch” in order to succeed. Furthermore, some learners struggled despite both levels of hints. In the videos, two of them turned to their neighbors and asked for the correct letters, and/or to chat. In some cases, participants were embarrassed to ask their neighbors for help again after so soon, and hence used the menu to skip to other levels in the games. More important, we observed that participants – especially among the low-gains learners – may be able to spell the words in the spelling activity, but were not able to spell the same words on the post-test. We offer two plausible explanations. Firstly, some children may have learned to spell the words by their last session in the program, but had forgotten their spellings between the last session and the post-test. Secondly, some participants never learned to spell the words in their entirety, since the spelling activity only involved filling a few blanks and did not require the learner to spell the entire word. Nowhere in the video recordings did we observe any child struggling with usability problems. X. CONCLUSION Our reactions to the results of the learning assessment were mixed. In an underdeveloped region where rural children did not have access to quality English instruction in their regular school or elsewhere, we were excited to see the participants -both high-gains and low-gains learners – in the after-school program exhibit statistically significant post-test gains that could be reasonably attributed to our cellphone-based English learning games. On the other hand, the learning benefits were uneven among participants. This could be a cause for concern. To begin with, high-gains learners outperformed low-gains participants on the pre-test, qualifying test and Hindi literacy test. In fact, participants’ post-test gains appeared to be highly correlated with their existing levels of spelling proficiency (as


measured by their performance on the spelling section of the qualifying test) and the grades in school that they are currently enrolled in. This observation suggests that those rural children with a stronger academic foundation are the same children who are most well positioned to take advantage of the benefits that cellphone-based learning confers. Our results are consistent with the outcomes of a study with rural and urban low-income children in India described in He et al. [16]. This study showed that weaker students benefited more from a teacher-directed pedagogical intervention, while stronger students benefited more from a self-paced, machinebased approach to English learning. These results should not, however, be interpreted to mean that we rule out technologyaugmented learning completely in the context of low-income children. Horowitz et al [15] reported a study on videos for learning the English alphabet streamed over cellphones. In this study, a greater proportion of lower-income parents, vis-àvis their higher-income counterparts, perceived the videos to have improved their children’s knowledge of the alphabet. In the face of the above overwhelming odds, what can we do to promote more equitable educational opportunities in the developing world? One possible – and perhaps cautiously optimistic – interpretation of the above results is that future research needs to be directed at understanding how e-learning software can provide more scaffolding support for those rural children who have less academic preparation. As an example, the spelling activity needs to be redesigned such that the learner is guided to spell the entire word eventually. With this redesign, however, gameplay becomes prolonged and can potentially increase player frustration, as we have witnessed above. One remedy is to have e-learning games track learner performance, so that the software can be adaptive in skipping stages that are similar to those that the player has previously performed well in. Another implication for instructional design, which calls for additional investigation, is scaffolds such as hints that are less conspicuous, so that their appearing on screen does not diminish the learner’s sense of self-esteem or achievement. Unfortunately, adaptive educational applications require the application state to be stored and retrieved on the same mobile device. From the logistics standpoint, this requirement is more difficult to implement in developing regions since it is harder to ensure that the same learner uses the same phone – which stores his performance from prior session(s) – across sessions. For instance, in an after-school program where attendance fluctuates from session to session, it would be prudent to keep a shared pool of cellphones, such that children who show up for the day’s session can draw from. In these circumstances in which it is not possible to reserve a cellphone for each child, a wireless networked mechanism for synchronizing application state across all cellphones may be necessary. This, and other issues that we have raised above, require further investigation for cellphone-based literacy learning to be more effective in targeting less academically advanced rural children. As we think more widely beyond the cellphone to consider it as a component in the broader learning environment, since

the after-school program is a model that is readily replicable, we encourage the reader to adopt and experiment with the lessons from this paper. Our results suggest that the cellphone – which remains a relatively scarce resource in the developing world – is most effectively utilized in an after-school program that targets more advanced children. This restriction may be a necessarily evil until we gain a deeper understanding of how to design instructional scaffolds for less well-prepared rural children. Next, children’s tendency to seek help from their neighbors can be channeled productively if the latter are taught to offer help appropriately (e.g. instead of only telling their neighbors the correct spelling, help them to associate and remember the correct spelling). Such peer coaching strategies are especially crucial since cooperative group learning is unfamiliar to many rural children, whose schools (if they attend one) are more likely to implement rote learning. Alternatively, such an afterschool program can hire facilitators to provide academically less prepared learners with similar coaching. ACKNOWLEDGMENT We thank Suraksha (Urvashi Sahni and Shalini Mathur) and Sesame Workshop India for collaborating with us on this pilot study. Lauren Bailey and Anuj Tewari provided much-needed administrative assistance, while Aman Anand and Siddhartha Lal provided technical assistance. We are especially indebted to Mehnaaz Abidi, Jatin Chaudhary, Neelima Purwar, Gautam Singh and Kavish Sinha for providing ground support. Lastly, we thank the parents and their children for their participation. REFERENCES
[1] J. Pal, M.Lakshmanan, and K. Toyama, “’My child will be respected:’ Parental perspectives on computers in rural India,” in Proc. of 2nd IEEE/ACM International Conference on Information and Communication Technologies and Development, Bangalore, India, Dec. 2007. S. Shukla, “From pre-colonial to post-colonial: Educational transitions in southern Asia.” in Economic and Political Weekly, 31(22), 1996, pp. 1344-49. D. Faust, and R. Nagar, “Politics of development in postcolonial India: English-medium education and social fracturing,” in Economic and Political Weekly, India, July 28, 2001. M. P. Kishwar, “Deprivations’s real language,” in The Indian Express, September 15, 2005. Available: http://www.indianexpress.com/printerFriendly/12662.html. Azim Premji Foundation, “The social context of elementary education in rural India,” 2004. Available: http://www.azimpremjifoundation.org/ downloads/TheSocialContextofElementaryEductaioninRuralIndia.pdf J. P. Gee, What Video Games Have to Teach Us About Learning and Literacy, Palgrave Macmillan, 2004. A. Banerjee, S. Cole, E. Duflo, and L. Lindon, “Remedying education: Evidence from two randomized experiments in India,” NBER Working Paper No. 11904, December 2005. S. Mitra, J. Tooley, P. Inamdar, and P. Dixon, “Improving English pronunciation: an automated instructional approach,” in Journal of Information Technologies and International Development, Vol. 1, No. 1, Fall 2003, pp. 75-84. M. B. Dias, G. A. Mills-Tettey, and J. Mertz, “The TechBridgeWorld initiative: Broadening perspectives in computing technology education





[6] [7]



and research,” in Proc. of the international symposium on Women and ICT: Creating Global Transformation, ACM Press, 2005. [10] B. Kothari, A. Pandey, and A. R. Chudgar, “Reading out of the ‘idiot box’: Same-language subtitling on television in India,” in Journal of Information Technologies and International Development, Vol. 2, No. 1, Fall 2004, pp. 23-44. [11] T. H. Brown, “The role of m-learning in the future of e-learning in Africa?,” in Proc. of 21st ICDE world conference, Jun. 2003. [12] M. Kam, A. Agarwal, A. Kumar, S. Lal, A. Mathur, A. Tewari, and J. Canny, “Designing e-learning games for rural children in India: A format for balancing learning with fun,” in Proc. of ACM international conference on Designing Interactive Systems, Cape Town, South Africa, Feb. 2008. [13] Y. M. Islam, Z. Rahman, S. S.Razzaq, M. A. Sayed, and S. Zaman, “Effect of feedback during lecture style delivery both in a face-to-face classroom and during a distance education television session in a developing country like Bangladesh without the use of Internet,” in Proc. of 6th IEEE international conference on Advanced Learning Technologies, pp. 469-471, 2006. [14] F. Librero, A. J. Ramos, A. I. Ranga, J. Triñona, and D. Lambert, “Uses of the cell phone for education in the Philippines and Mongolia,” in Distance Education, vol. 28, no. 2, pp. 231-244, Aug. 2007. [15] J. E. Horowitz, L. D. Sosenko, J. L. S. Hoffman, J. Ziobrowski, A. Tafoya, A. Haagenson, and S. Hahn, “Evaluation of the PBS Ready to Learn cell phone study: Learning letters with Elmo,” reported prepared by WestEd, Sep. 2006. [16] F. He, L. L. Linden, and M. MacLeod, “How to teach English in India: Testing the relative productivity of instruction methods within the Pratham English language education program,” working paper, Jul. 1, 2008. [17] U.S. Pawar, J. Pal, R. Gupta, and K. Toyama, “Multiple mice for retention tasks in disadvantaged schools,” in Proc. of ACM conference on Human Factors in Computing Systems, San Jose, CA, Apr. 2007. [18] N. Moraveji, T. Kim, J. Ge, U. S. Pawar, K. Inkpen, and K. Mulcahy, “Mischief: Supporting remote teaching in developing regions,” Systems, pp. 753-762, Florence, Italy, Apr. 2008.


Kelsa+: Digital Literacy for Low-Income Office Workers
Aishwarya Lakshmi Ratan1, Sambit Satpathy2, Lilian Zia3, Kentaro Toyama1, Sean Blagsvedt4, Udai Singh Pawar5, Thanuja Subramaniam6
menial labor in the same geography. In India, for example, the ratio of support staff to direct employees can range from 10% to as high as 60% – 30 support staff for an employee base of 100 is not unusual. Given that the corporate IT and ITES sector alone employs more than 1.6 million professionals in urban India [1], the support staff at these facilities would involve a population of at least 160,000 and more likely 320,000 urban low-income workers. Such workers typically have limited or low-quality education and earn between $50 and $200 per month. They have few on-the-job opportunities to upgrade their skills or learn new ones, and therefore remain caught in a vicious cycle of low-income work. We propose Kelsa+, a program that offers Internetconnected PCs for free, unrestricted use to the low-income workers in modern offices in developing countries. In our 18month pilot, we tried several things to encourage usage and improve development impact, and recorded a variety of positive, if limited, results. Kelsa+ could thus be a worthwhile project to spread to other offices in developing countries, if these benefits could be delivered more systematically. In this paper, we report on our pilot experiment with Kelsa+ in Bangalore, India. We discuss our design decisions, describe usage patterns, and identify explicit development-focused opportunities and outcomes. We also discuss some of the potential challenges to adoption by other organizations. Although PCs are likely to have been made available to service staff elsewhere, perhaps on an informal basis, to our knowledge, a deliberate exploration of the design and value of such a project has never been conducted before. II. RELATED WORK There are three threads of research that are relevant to the Kelsa+ study: the Hole-in-the-Wall experiments for children, computer-aided learning for adults, and computer kiosk or telecentre initiatives aimed at promoting development in poor communities. The ‘Hole in the Wall’ (HitW) experiments The National Institute for Information Technology (NIIT) [2] ran a series of computer-based education experiments with children from disadvantaged communities in New Delhi, India, in the late 90s. Effectively, they bore a hole in the wall that separated NIIT from the neighboring slum settlement and had an Internet-connected PC set up facing the settlement, with a touchpad built into the wall for navigation. They found that children from the slum communities in the vicinity

Abstract—Almost all formal organizations employ service staff for tasks such as housekeeping, security, maintenance, and transport at their office facility. Many of these workers earn wages in line with menial-labor salaries in their respective countries. They have few onthe-job opportunities to upgrade their skills or learn new ones. Kelsa+ is an initiative through which organizations in developing countries can increase digital literacy and skill development among such low-income workers, through the provision of an Internetconnected PC for the service staff’s free, unrestricted use when off duty. We study a Kelsa+ pilot implementation in Bangalore, India, involving an office facility with 35 service staff. In a preliminary exploration over 18 months, we find that at a cost that is negligible for the organization, workers’ use of the Kelsa+ PC is high and can deliver benefits both to themselves and to the office. For workers, broad gains were seen in confidence, self-esteem, and basic digital literacy, while a few individuals experienced improvements in second-language (English) proficiency and career opportunities. These early results point in the direction of a cost-effective ICT4D initiative that could be run in the developing-country offices of the very organizations promoting development off-site. Index Terms—ICTD, digital literacy, service staff, low-income workers, urban poor

I. INTRODUCTION Those of us working in “information and communication technologies for development” (ICTD) often run projects in remote rural areas or urban slums to work with low-income communities. Meanwhile, we often neglect a low-income group right under our very noses: workers who clean our offices, provide security, maintain facilities, etc. Most offices involve a sizeable group of service staff, who take care of the housekeeping, security, transport, maintenance and so forth of the facility. In developing countries, service staff tends to be employed in large numbers, with many of them earning wages equivalent to those for
Manuscript received September 22, 2008. Revised February 20, 2009. 1 Aishwarya Lakshmi Ratan and Kentaro Toyama are with Microsoft Research India, “Scientia”, 196/36, 2nd main road, Sadashivnagar, Bangalore – 560080, INDIA. (Corresponding author: Aishwarya Lakshmi Ratan - phone: +91-80-66586000; fax: +91-80- 23614657; email: aratan@microsoft.com) 2 Sambit Satpathy is with CISCO 3 Lilian Zia is with INTEL 4 Sean Blagsvedt is with Babajob.com 5 Udai Singh Pawar was with Microsoft Research India 6 Thanuja Subramaniam is with Microsoft, Sri Lanka


(mostly 6 to 14 years old) explored PC usage on their own using the device, resulting in significant learning benefits without any formal instruction. HitW interventions for children have been conducted at various rural and urban sites, and the pattern of improved digital literacy from this Minimally-Invasive Education (MIE) intervention has been consistently recorded [3]. Critiques of the project have come from those who see the “digital divide” not as “gaps to be overcome by providing equipment,” but as “social-development challenges to be addressed through the effective integration of technology into communities, institutions, and societies” [4]. Such critics find the HitW intervention to be “technologically deterministic” and lacking in community and parental involvement. Further, they question whether simply having physical access to an IT device and learning a set of basic IT skills would translate into any systematic improvement in the lives of these children. Kelsa+ was initially inspired by the ‘Hole in the Wall’ project. It examines unrestricted PC access not for children, but for adults, specifically low-income urban workers, though within an institutional and social context. Computer-Aided Learning for Adults Computer-Aided Learning (CAL) has existed in various forms for a number of decades as an educational tool in schools [5], universities [6], healthcare institutions, and is becoming an important area for ICTD investigation in developing country contexts [7]. Early experiments in basic computer-assisted instruction, comprehensive historical overviews of which have been done [8], has led to more sophisticated recent work around using robotics for science education [9], facilitated distance learning [10], and so on. However, there have been relatively few investigations around uptake and impact in the context of CAL for adult learning. One study identifies the importance of CAL in maintaining the continuity of learning history of adults, arguing that such use of computers in learning will be able to shift the focus of cognitive energy to cognitive creativity, allowing adults more space for absorption of knowledge and creativity [11]. Kelsa+ examines a subset of questions in the domain of CAL for low-income urban workers in developing countries. Computer kiosk and telecentre ICTD initiatives Most of the adult digital inclusion projects in the developing world have been undertaken as computer telecentre initiatives [12]. These typically, though not always, involve a small number of PCs set up in a rural area, offering a variety of PC-related services and run by an entrepreneur, salaried operator, or a community-based organization. The expectation in these projects is often that once ICTs such as the PC are made physically available in communities where they did not exist before, there will be usage by members of the community and socio-economic development will follow. This approach is evident in the project concept notes and promotional material of several telecentre initiatives: “by providing information about employment, better farming techniques and health we hoped for new sustainable job

opportunities, improved farming knowledge and healthier life.” [13] However, many of these projects have found it difficult to sustain the technology and to establish clear links between ICT access and development impact. In many instances, usage of the facility (and therefore revenues in for-profit kiosks) falls over time, irregular connectivity in remote areas causes periodic interruptions in service provision (and further decline in usage), device maintenance accrues unanticipated costs and where high usage persists, it is often for the usage of applications with limited welfare impact and by community members who are not particularly disadvantaged (making it hard to justify public expenditure) [14],[15]. Many programs face a tension between pursuing financial sustainability and maximizing social outreach [16]. Those kiosks that are able to sustain usage often involve an exceptional kiosk operator or a committed organization that is able to effectively coordinate between the end-users and the desired application [17]. In contrast, Kelsa+ operates in a modern office environment, where exactly the elements for sustainability are already present – IT maintenance and support, good physical, electrical, and connectivity infrastructure, caring staff, etc. A few kiosk projects, such as the Akshaya project in Kerala, have explicitly pursued adult digital literacy as an objective [16], [18]. Despite the project’s impressive scale, however, field studies suggest that these programs achieved neither the reach nor the depth of digital literacy that was sought [28]. We hypothesize that one reason for this was that digital literacy courses were conducted over very short time periods (10 sessions of 90 minutes each), which did not permit learners enough time to familiarize themselves with the technology. In Kelsa+, interaction with the PC is voluntary and continuous in a process we call “digital habituation” [19]. This allows the incremental build-up of digital literacy skills at a user-determined pace, and through user-determined content and applications. III. BASELINE INVESTIGATION Low-income urban workers largely live in the city’s less developed residential settlements with many working as part of the informal economy. A large share of these workers are young and recent urban migrants. Given low levels of education or low-quality education, they are mostly employed in low-paying service sector jobs or work as small entrepreneurs. Among this group, some workers find jobs at formal office facilities, often associated with slightly higher pay, additional perks, and more prestige. In Indian cities, these opportunities have mushroomed since the early 1990s, as both domestic and multinational corporations expanded operations and facilities. Unfortunately, these workers have limited opportunities to upgrade skills or learn new ones, especially within their workplaces. The training institutes that exist, for spoken English, typing, or IT skills, place heavy demands on workers’ time and finances, as they require attendance at external training centres. As a result, even after decades of labor,


workers earn only marginally better incomes than when they started. Our pilot focused on the service staff at a single urban corporate facility in India’s IT capital, Bangalore. The facility employs around 55 full-time employees and 35 support staff at any given point in time. The latter group consists of housekeeping or cleaning staff, drivers for the office cabs, security guards stationed at the facility’s entry and exit points, and building maintenance staff to monitor the facility’s electricity, connectivity and other infrastructure. Prior to the pilot, we conducted detailed structured interviews with nearly all of the facility’s support staff (a sample of 30 respondents out of a total group of 35 workers), to understand their baseline socio-economic characteristics, occupational history, current job requirements, use of technology and aspirations. Each interview lasted for 60-90 minutes. This was complemented with participatory observation of work routines. Of the many insights gained from this qualitative investigation, a few are described below. The average age of the support staff is 26 years, and the average worker had ~12 years of formal schooling (class 10). All except two are men: many have migrated to the city alone in search of better work opportunities, even as their families continue to live in the village. The average worker earns ~$100 in income per month [20]. Depending on their job and the week in question, workers’ shifts varied: security guards and maintenance staff rotated for a week each between a 9 pm – 7 am shift, 7 am – 2 pm shift, and a 2 pm – 9 pm shift; housekeeping staff rotated between 9 am – 4 pm and 12 pm – 9 pm shifts; and drivers worked 12 hour shifts from 10 am 10 pm and 10 pm - 10 am. Workers had changed numerous jobs, despite having entered the labor force recently. One worker for instance had started out in the village doing casual wage-labor work, then worked as a private tailor, moved on to work at an export garments factory, then changed jobs to work at a finance company, after which he moved to corporate housekeeping. In the course of their daily work at the office, workers regularly saw PCs but did not touch or interact with them, except to clean them. Their interactions were exclusively with the specific ‘tools of their trade’, involving coffee machines, vacuum cleaners, fax and photocopying machines, radios (for the drivers), and phones. A few workers had occasional interactions with a PC at a cyber café, or at the office for a specific application, e.g., managing the Building Management System or the Security Camera Tracking application. Such workers’ general digital literacy skills were very low though – a result of their repeated restricted interaction with a single niche application. Likely because they worked in an office full of PCs (a software development and research centre), PCs were dominant in the discourse of the workers’ aspirations for themselves and their children. On a four-point Likert scale (‘Not At All Important’ to ‘Very Important’), all except two respondents rated the computer as being either Important or Very Important for their own upward mobility, and all of them felt this way about the importance of computers for their children to get ahead in life. As one respondent claimed,

“Even if you are poor, if you learn computers and try and get used to it - you can improve” [21]. Yet, though considered critical at a conceptual level, functional understanding of the PC was limited and based on workers’ observations from mass media and their environment. One worker’s comments on the PC as a learning device aptly captures this lack of a clear functional understanding, “Students now learn Windows - how to open it and use it, what’s inside it, games, etc. - and I can’t say why exactly that is useful, but it is. I know there is something in the computer that is important for students” [21]. A few workers who had learned to use particular applications meaningfully on a PC had done so through instruction from their peers or seniors at a past workplace. They described having picked up these skills through a combination of observation and ‘learning by doing’. As one respondent described, “I learned to use Outlook from my boss here. I had learned to use Excel at my previous job at a travel agency – a lady colleague taught me there. I learned to browse the internet after observing how my friends did it. I learned to do personal email on my own.” IV. SOLUTION: KELSA+ In response to the expressed desires in our baseline qualitative study, we introduced an Internet-connected PC at the workplace for the exclusive and free use of the facility’s support staff, to be used during workers’ off-duty hours. The project was named Kelsa+, with ‘kelsa’ being the local language (Kannada) word for ‘work’. Kelsa+ was, therefore, meant to signify ‘after-work’, ‘beyond-work’ and ‘improvingwork’. The intervention involved three overlapping phases of activity: (1) a ‘Hole-in-the-Wall’ phase inspired by the NIIT studies, when we were just observing what workers did on the PC when left on their own, (2) a Learning Modules phase, which involved trying different things to improve productive value of the PC for the workers, and (3) a Pre-Expansion phase, when we conducted interviews with management at other firms to understand what issues we’d need to address to make Kelsa+ work in different office locations. We describe each of these below. A) Phase One: An Office ‘Hole in the Wall’ The first phase involved establishing the four basic components of the project. First, a worker-dedicated PC was set up and integrated with existing company structures to have full infrastructural and institutional support. The PC had a basic Windows XP Operating System and the Office 2007 suite of applications installed. It had a dedicated broadband internet connection, as well as peripherals such as speakers, headphones, a printer and a webcam. User log-in on this PC was disabled to minimize barriers to entry, so that a worker could begin interacting with the PC as soon as s/he sat in front of it. That the maintenance of this PC would be handled by the company’s regular IT staff was also established. Second, the Kelsa+ PC was set up in the office’s basement,–a space that housed the maintenance office, the


workers’ changing rooms, the office cars, etc, and so was shared by all the service staff. The fact that it was placed in a location that was primarily “owned” by the workers and in which they felt comfortable was deliberate. Had the PC been placed in the cafeteria or in the lobby of the building, workers might have used the facility less, in deference to the office’s full-time staff. In some ways, this placement choice of using a space that the target group felt comfortable accessing was probably just as important as the ‘Hole in the Wall’ project's decision to place their kiosks near schools and outside of buildings. As a worker later affirmed, “When the computers are inside, it is difficult for us to feel comfortable touching them. Now, this is like a computer for us, outside [in the basement]” [20]. Third, in a set of initiation activities, the PC was introduced to the workers as a facility that they could use for any purpose whenever they were off duty. Workers were informed that all activities on and around the PC would be recorded, both for research and security purposes, specifying that none of this material would ever be used to restrict usage. Signage in the local language and English indicating this logging was also placed at the Kelsa+ PC location. For any questions or concerns, two of the authors were pointed out to the workers as the people to contact. Finally, the determination of who would use the PC when and how was left entirely to informal processes of negotiation between workers. No schedules were drawn up. No instructions were given. That usage and learning was entirely voluntary and self-paced was established. No restrictions were laid on how many users could simultaneously work on the PC. The intent was to provide a wide open area, conducive to groups of peers interacting with the PC together. B) Phase Two: Learning Modules After a year of allowing unrestricted usage of the PC, we collected feedback from the workers in a series of focus group discussions. Strong interest was displayed in achieving particular kinds of learning using the PC, including learning English, office productivity software, and accounting software (Tally). To further this goal, we started by introducing English as a Second Language (ESL) content, both as CDs and compiled weblinks, to the workers. No formal instruction was given or classes were taken. The voluntary and self-paced learning nature of phase one continued. We only acted as initiators of introducing new content at the Kelsa+ PC towards a specific productive goal, and not as evaluators or routine instructors. C) Phase Three: Management Discussions Kelsa+ is housed within an institutional context, and we made an effort to understand the effect that the intervention was having on its social and institutional environment, to avoid some of the limitations that the Hole-in-the-Wall study had faced. As we continued with other explorations on Kelsa+, we collected feedback from management to understand their reactions to the Kelsa+ facility for workers. We also began a set of investigations to understand what it would take to expand the project from a small-scale pilot at

one office location, to multiple sustained deployments across various firms. For this, the project was consolidated as a ‘Corporate Social Responsibility’ initiative, and proposed to a number of major corporations in Bangalore. Senior management at these firms were interviewed in relation to their own service staff populations. This data was used to compile a variety of solutions to the expressed concerns, so that the project could possibly fit into other institutional contexts. V. STUDY METHODOLOGY We employed a mixture of research methods in studying Kelsa+ through its three phases of activity, which we describe below. Collective usage: quantitative metrics The Kelsa+ PC had a logging application installed, which tracked all events initiated on the PC [22]. This included launches of all applications, as well as URLs visited. The logging tool also allowed an examination of how much time users had spent on the Kelsa+ PC on any given day. These three metrics (collective time spent using the machine, applications used, websites visited) were recorded over the 18-month period (Jan 11, 2007 – June 2 2008), with the analysis conducted on consecutive fortnights of activity. Application and internet usage was measured using number of launches as well as active time spent on each application/website. There were two breaks in logging, one when the PC’s connectivity was interrupted in mid-November 2007 and the other in late-March 2008 when the PC’s OS was re-installed, both in response to virus attacks. The categorization of the log data into meaningful groups, including application categories and website categories, was performed based on the researchers’ examination of keywords that tagged the data optimally despite the heterogeneity of usage. These are available on request from the authors. Collective usage: qualitative measures The activities on the Kelsa+ PC were also recorded using a screenshot logger, with a screenshot of the PC taken every minute. In addition, the activities around the PC were recorded using a motion-detecting web camera. These screenshot and video logs were processed manually, and hence, selectively, to answer specific questions around usability, group usage/sharing dynamics and number of distinct users. For the latter, 14 days of video logs were randomly picked from the 18-month study period to estimate the average number of distinct users per day. Individual impact: quantitative metrics of change Given the high job turnover rate among workers in this segment (described in the baseline study’s results), longitudinal tracking, i.e. measurements for the same individual over time, was very challenging. However, for a limited subset of workers, we obtained before and after results for particular behavioral or proficiency tests and these cases


were used to understand possible patterns of change. These included the following tests: • A Self-Esteem questionnaire: A local languagetranslation of the Rosenberg Self-Esteem questionnaire [23] was administered to a sub-set of workers prior to the Kelsa+ intervention (17 workers), and then five weeks after it was introduced (27 workers). Eleven respondents overlapped between the two administrations (0.65 of pre-test group). The differential in scores obtained for these workers was calculated. • A Digital Literacy test: A brief digital literacy test covering usage of basic applications on the PC was administered prior to the introduction of the Kelsa+ PC (30 workers). The time taken to complete a task, as well as the number of prompts needed to complete the task, were recorded as metrics of proficiency. The same test was administered to three workers, 18 months after the intervention, of whom two had taken the pre-test as well. The differential in scores obtained for these workers was calculated. Normal staff turnover at the rate of ~2 workers per month had disallowed a pre- and post-test comparison for more workers over this extended time period. • An English proficiency test: An adapted version of Cambridge University’s ‘Key English Test’ for Beginners [24] was administered to a sub-set of workers first in March 2008 (20 workers), prior to the introduction of the ESL digital content. A second test in the same format was administered in August 2008 (17 workers), after over three months of the ESL material having been available for workers’ usage. The questions focused on testing basic English vocabulary, grammar, and reading comprehension. Seven respondents overlapped between the two administrations (0.35 share of pre-test group). The differential in scores obtained for these workers was calculated. The post-test also asked a few questions on workers’ Kelsa+ usage or non-usage over the week preceding the test. Individual impact: qualitative narratives of change A subset of the staff from across workgroups (6 workers, ~0.17 share) was interviewed in detail 18 months after the intervention was launched. These workers discussed their background, prior exposure to a PC or not, their usage of the Kelsa+ facility, and their present job, following a structured interview protocol. Their perceptions of change since the introduction of the Kelsa+ PC were also recorded. The results from this analysis describe socio-economic mobility pathways associated with usage of the Kelsa+ PC. Management interviews We conducted structured interviews with five facility managers at the pilot location (across transport, security, housekeeping, etc.) and recorded their observations around the Kelsa+ intervention including: the usage patterns of workers, any effects on worker productivity (positive or negative), any other changes in service staff behavior or work since PC access was given, etc.

To assess the relevance of Kelsa+ in other institutional contexts, over 15 corporations were contacted from a range of sectors (biotechnology, energy, IT) with the majority being IT firms. They were first briefed about the Kelsa+ concept. Structured interviews were then conducted with them in which they shared details on the size and composition of service staff at their facilities, their evaluation of the possible benefits from, as well as concerns around, implementing Kelsa+ as a CSR initiative for the service staff at their facility. VI. STUDY RESULTS Given the multiple research methods that were used, in this section we present our results in correspondence to thehighlevel categories described in the methods section: collective usage,individual impact and management feedback. VI.I Collective Usage Time used After the initial fortnights, the Kelsa+ PC saw high and sustained adoption by the support staff (Figure 1), recording average collective usage of 10.13 hours per day. Usage peaked at 17.83 hours per day in fortnight 29. Further, usage was not restricted to a small minority of persistent users, but was spread across a broad base of workers. From the video log analysis, we found an average of 13 workers to be primary users of the PC each day, with all staff workgroups represented in the user base. 40 unique workers were observed using the Kelsa+ PC at least once (despite a staff size of only 35 at any given point in time – this occurred due to staff turnover) during the sampled period. A quarter of the workers were identified as high-frequency users, seen using the PC on half or more of the observed days [19].

Figure 1: Hours of active usage of the Kelsa+ PC per day (average over each fortnight) Using a different measurement instrument, the responses to the questions on Kelsa+ usage or non-usage included in the second English proficiency test revealed that all but two workers in the sample (0.88 share) were active users of the


Kelsa+ PC. They reported using it on an average of 4 days in the week preceding the test, and for an average of 60 minutes per sitting.

Usage dynamics The applications and content accessed on the Kelsa+ PC changed over the study period. Figures 6 and 7 (on pp. 13) showcase the changing distribution of application and internet usage over 31 fortnights. Initial application usage was spread evenly across the gamut of available software on the PC, indicating heavy exploratory usage. However, starting in the third fortnight, Internet Explorer becomes the dominant application launched, followed closely by multimedia applications. There is a shift away from basic interactive applications that are very popular initially (Microsoft Paint and offline games, whose usage fell from 15% to 0), to increased usage of the Internet with its dynamic content and more sophisticated applications. As Figure 7 shows, workers spent many months primarily using the internet for entertainment – viewing music videos or films on YouTube or Stage6. However, we see a rise in using the Internet for communication, through email and social networking sites like Orkut, eight months after the project was started (correlated with 6 workers simultaneously creating email accounts at this time). There is a strong motivation to use the computers for selfexpression, as Figure 3 shows. The ability to create a personal digital presence, both through images and later through email, was a source of great pride among workers, and seemingly altered the way they perceived themselves. Learning Workers followed individual and group learning paths. Several workers simply observed their peers using the PC for weeks before attempting to touch the PC themselves. As one worker commented, “For the first one and half months, I just watched how other people used the computer.” We asked why he did this without trying to use the PC himself, to which he responded that “what if something happened when I used it?” [19] Observation and individual exploration established ‘learning by doing’ routines, which were instrumental to meaningful PC literacy gains.

Group usage of the Kelsa+ PC was very popular; in fact, some users actively sought out colleagues with whom to use the PC. Peer learning sometimes took the form of an informal teacher-student relationship for a particular task or application. For instance, as [19] describe, “many workers had gone through the email account creation process with a peer who was an existing email user. During the registration process, in the space where a second email address was required (usually of the person creating the account), the experienced email user friend would enter his email address, since for most workers, this was their first email account.” For other workers, group usage involved a symbiotic learning relationship: “I use it with a friend generally, so that we can learn from one another. What he doesn’t know I tell him, and vice versa.”

Figure 3: Workers creating personalized desktop backgrounds using the Kelsa+ PC’s webcam Usability preferences Given their educational and linguistic background, workers’ revealed a number of adaptations in input techniques to achieve functional use of the PC. Navigation using a mouse dominates since typing is problematic. Browser history is repeatedly used as an easy way to access content of choice, indicating implicit peer learning [Figure 4]. Search queries (with the “.com” tag) are repeatedly used to access online content within and across sites, given that it involves only memorizing and typing in one keyword or a short phrase, as opposed to an entire url. Email forwarding is the dominant method of online communication, with one user forwarding up to 15 emails per day to 10 or so people (mostly images).

Figure 2: Combination of workers observing and being primary users themselves


Figure 4: Browser history is repeatedly used as a way to access content without typing out URLs

VI.II Individual Impact A) Self-esteem and confidence gains The 11 workers who took the self-esteem test just before the Kelsa+ intervention and five weeks after its installation, recorded a 5% jump in their self-esteem score. This change clearly cannot be attributed just to the PC installation without a control group. However, qualitative interview responses from workers appeared to reiterate the effect that the Kelsa+ PC had had on individuals’ sense of hope, confidence and self-esteem. One driver commented, “I see some changes in my life – I couldn’t speak English before, now I can speak a little… I haven’t yet changed my job or used English outside, but I now have some hope to learn… I have that courage.” Further, use of the Kelsa+ PC was seen as a first step towards further possibilities: “To do any computer course outside, I think I should know some basics.... having picked up some basics today I have the confidence that when I take up a course... I will be able to catch up ... and I am very inspired by using the PC here to know more and learn.” Given that the Kelsa+ PC was in many cases the first time a worker had touched a PC, it elicited responses such as: “I felt so happy that day when we had the interview. For the first time I touched a computer and did so many things without a mistake….. I don’t have an email account. So now maybe I can make one on this computer.” [20] The continuous access to a PC at a location that workers visited everyday lowered the barrier to usage considerably, “Since, the computer is here, we get awareness! Also because we can see the computer daily ....my desire to learn and use it has increased very much.” And all this of course translated into changes in how workers viewed their workplace: “In all my service, this is one of the best workplaces I have seen.” B) Digital literacy gains Two workers, a housekeeping worker and a driver, with minimal prior PC exposure and varied demographic characteristics, were administered PC literacy tests before the

Kelsa+ intervention and 18 months after it was initiated. Churn in workers and the small sample size, curbed such a pre- and post-test for a larger share of workers. Such testing would be essential to establishing average impact across workers. However, the results for these two respondents are encouraging, given their low educational attainment and given that in these cases we can clearly trace the digital literacy skill gains to workers’ Kelsa+ PC usage alone. Both workers had experienced gains in a core set of PC literacy skills (see Table 1). While they were unable to perform any of the tasks in the pre-test (save for turning on the PC), both were now able to open a web browser window, launch a search application, navigate through the results, and open and close a document-processing application. Similar to the Hole-in-the-Wall gains for children, these measured improvements in digital literacy for two such demographically and occupationally distinct workers, from entirely voluntary and informal learning processes, are promising. Neither of these workers had interacted with a PC outside of the office during this time. Neither of them had been schooled in English. One of them had left school after class 7, the other after class 4. This only goes to show how despite the average educational attainment among the pilot location’s service staff group being higher at Class 10, even those who had much lower levels of education in a local language saw gains in basic digital literacy from Kelsa+ usage. Table 1: PC Literacy Test Results A R Occupation/ Age Driver/ 55 Housekeeping/ 23 Education/ Class 4/ Tamil Class 7 / Kannada Medium of instruction Annual Income $1600 $1000 (US$) Previous PC Never touched a Had touched a PC Exposure PC before, but never used one Reported usage 30-45 min each 30 min every day frequency session; 3-4 sessions a week Key applications Local-language Local-language used news portals, music and films, games, localemail, games language music and films PC literacy scores PrePost-18 PrePost-18 mths mths Turn on the PC ? Play Windows X X (offline) Games Open Internet X X Browser and go to a search engine Enter a search X ? X query Open the best X X


search result and go back to results list Sign-in to E-mail X X X client Start MS-Word X X ? application Type in MS-Word X X X X Save document X X X Print document X X X Close Word X X application Shut down PC X X : Successfully completed the task ? : Partially fulfilled the task/ completed the task with extensive prompting X : Does not know/ did not attempt Source: [19] C) Improvements in English proficiency Over a 3-month period of Kelsa+ usage, the English proficiency scores for the seven workers who took both preand post-tests improved from 32% to 41%. Looking at their individual trajectories, we find that of the seven, four saw increases in their English proficiency while the other three saw declines. Yet, the increments upward (average movement of +19.4%, p=0.04) appear to be significant while the movements downward are not distinguishable from measurement error of the same proficiency level (average movement of -3.8%, p=0.13). This possibly indicates an interaction between individual worker motivation or initiative and the availability of the PC-based learning material, to produce differential skill gain outcomes. A key issue is, therefore, the need to understand motivational tools that might make available skill-development tools more widely used across all workers.

high motivation levels, and their experiences are anecdotal. At best, their outcomes indicate the extent of welfare impact that is possible when worker initiative and circumstance interact with access to the Kelsa+ facility to produce welfareimproving outcomes. Upward mobility within the same office A housekeeping worker with a Class 7, local-language, rural government school education, and no prior exposure to a PC, began using the Kelsa+ PC for 60-90 minutes each day after his work shift. He used it for games, internet video, music, etc. A few months later, he created an email ID with the help of a driver colleague, began using the Learn-English CDs available at the Kelsa+ PC, and various Microsoft Office applications. He was given an initial typing job on MS Excel by the IT staff in the office, which he completed and submitted satisfactorily. He was then given additional inventory data entry jobs (on Excel), before being hired as a dedicated worker for the IT staff. English skill acquisition A driver with no prior exposure to a PC and a Kannadamedium government school Class 10 education, took an interest in using the PC for music, movies, as well as learning English, spending 2-8 hours of off-duty time per workshift on the Kelsa+ PC. He specifically requested an extension to stay in this job for a few more months (a number of drivers left the facility due to inconvenient shift changes at this time; contract workers are often moved from one location to another), since he had begun using the Learn-English CDs at the Kelsa+ PC and wanted to acquire a certain level of competence before having to leave. Shift to a new career path A security guard with a Class 12, local-language, rural government school education, sat at the Kelsa+ PC for 60-120 minutes before or after a shift. He used the Kelsa+ PC to practice typing as he underwent a data-entry training course outside the office. The practice afforded by access to the PC at the workplace was key, in his opinion, to his successful job interview for a data-entry job. He has since moved from his security guard job to a data entry position at another firm. His pride in his new job was reflected in his comments, “Today I can stand up in front of my father and friends and say that I am no more a watchman, but I am doing a computer job."

Table 2: English Proficiency Test Results Overall Test 1 Test 2 Average score (%) 45% 53% +8% Sample size (n) 20 17 Pre- and post-test cases (7 workers) Average score (%) 31.7%


+9.5% (p=0.14)

VI.III Management feedback A) Managers at pilot location The response from the management at the pilot location was strongly positive. They felt that such exposure to technology made their workers confident and knowledgeable, while also being a source of recreation. Moreover, they commented that providing a PC access facility for the service staff at the workplace had a special connotation for workers: “sitting in a cyber [café] and learning and sitting in their office and learning is something different.”

D) Socio-economic mobility pathways While the measured estimates may indicate gains in particular skills for select individuals, the true impact of such facilities would need to manifest as improved socio-economic outcomes for workers. Three cases illustrate the variety of ways in which these skill gains have translated into individual workers’ ‘development’. These workers displayed particularly


Certain improvements in the workers' ability to communicate in English were described. In addition, some workers had demonstrated initiative in integrating PC usage into their own everyday workflows. A worker in the housekeeping division, for instance, had begun to type up the weekly stock order list for the office pantries on MS Excel and was emailing these to his supervisor, instead of writing them out by hand (Figure 5).

location for the PCs due to safety measures in some companies, risk of downloading or sending malicious or offensive content from the PCs, legal concerns regarding company liability, the risk of computers being used for entertainment purposes only, and the associated effort for regulation and supervision of usage being too huge for companies to undertake. [25] The fact that most of these workers are contracted and are not direct employees of these companies also contributed to reluctance by some corporations: “These workers are not our employees so why should we educate them. Their employers may not like it.” In general cost was not perceived to be an obstacle in proceeding with Kelsa+ amongst most of the companies interviewed. Connectivity charges were considered the biggest cost component, but procuring PCs for the initiative was not perceived to be an issue. VII DISCUSSION

In proposing Kelsa+ as a potentially effective ICT4D intervention for low-income office service staff, we return to the related work in this field and discuss this intervention’s comparative strengths and limitations. (1) Extending ‘Hole-in-the-Wall’ possibilities Similar to the HitW intervention, Kelsa+ highlights how acquiring basic digital literacy skills has little to do with socio-economic background or educational quality, but much more to do with immersion, practice, and learning by doing. Persistent and minimally-invasive access to the PC for workers’ continuous exploration in the office environment through Kelsa+ has allowed for the following primary benefits: (a) Improvements in basic digital literacy, including the ability to turn a PC on, operate an input device effectively, identify various applications, and navigate through the web to preferred content. (b) Improvements in English proficiency, driven not only by the use of dedicated ESL content, but also by the workers’ repeated interaction with English language content on the web, and sporadic use of email and office productivity tools. (c) Improvements in hope, confidence and self-esteem. Kelsa+’s achievement lies in its demonstration of how easy PC usage can be, and its encouragement to workers to believe that they can learn more as they work and at no additional cost. Should this hope translate systematically into real outcomes manifested in test scores, or new jobs, workers will be further inspired to experiment and learn, thereby establishing a virtuous feedback loop. In a departure from the HitW studies, Kelsa+ also explores the effects of the intervention on the institutional context that houses such an intervention. The pilot office facility saw greater staff morale and attachment to the workplace from having such a recreation and learning facility at the

Figure 5 a and b: Housekeeping workers maintaining task lists on MS Excel at the Kelsa+ PC (even as they multitask with playing games) B) Managers at potential implementation locations The feedback from the fifteen corporations contacted as potential implementers of Kelsa+ facilities fell into two camps. One set of corporations saw Kelsa+ as a tool for their workers to learn English, which they considered a major factor in increasing workers’ confidence and employability. The intervention would bring more equity and awareness to their service staff, and change the way they viewed technology and the world. As one manager commented, Kelsa+ is "not just a little feel good project, but something that can have an impact if done right.” [25] On the flip side, the managers at a number of other corporate facilities were very concerned about the potential risks of this intervention. Security of both physical equipment and content was perceived as the number one concern. Other perceived risks included difficulty in finding a suitable


office. Worker initiative to incorporate the Kelsa+ PC into their work routines was welcomed. Managers responded positively to improvements in worker confidence and knowledge, and to enhanced worker capacity through improved digital literacy and communication skills. At the same time, it is unclear whether these dynamics of security, productivity, and open exchange seen in the pilot will repeat themselves in other Kelsa+ locations. For instance, restrictions in the timings that workers can stay at the office facility, before or after their workshifts, will indeed influence the extent to which they can experiment with the PC and gain from access. Resolving such issues will involve political bargaining between stakeholders, a process inherent to the effective design and implementation of any sustainable ICTD intervention [26]. (2) Slow and sustained wins the race What makes the Kelsa+ installation any different from other telecentre/ kiosk or digital literacy initiatives in developing countries? There are three key aspects of the Kelsa+ system’s design that distinguish the user experience it offers lowincome service workers from other similar ICTD initiatives: (a) Availability and support: The office environment provides 24x7 availability and all the necessary infrastructural, technical, social and financial support required to sustain the health of the PC for the workers. Skilled maintenance is always at hand, more experienced users are all around to ask for advice, and the institution is already an integral part of workers’ daily routines. As a result, the barrier for workers to begin experimenting with the PC is considerably lowered. (b) Digital habituation: Interaction with the Kelsa+ PC has no fee and is not limited in any sense. This allows workers to respond to the PC spontaneously and evolve behaviors incrementally as they build skill. Workers engage in a slow process of familiarization with the technology as they learn basic navigation techniques and understand what the technology can be used to achieve (or not) and how; a process [19] term ‘digital habituation’. Such habituation “constitutes a critical intermediate step between providing PC access to a disadvantaged community and achieving sustained development impact.” [19] (c) Learning by doing / learning through peers: An advantage offered by the Kelsa+ PC and usually not available at regular telecentres, formal IT training institutes, or libraries, is the ability to learn passively and informally from peers, in addition to learning actively through doing. As a security guard who had done two years of computer courses at a government college noted: “They used to teach us basics...but I didn’t pick up much from the class… I generally learnt things on my own after coming here...once I see some people using [certain applications or features] ....and then next time I generally go about following the same.”

(3) Low marginal cost for well-endowed providers In the PC kiosk franchise arrangement, there is often an entire livelihood, that of the kiosk operator, depending on the success or failure of the PC kiosk business. In contrast, one of the major advantages of Kelsa+ is that it is an intervention that costs marginally nothing to those financing and running it, i.e. large private corporations. The providers in this case already manage large IT budgets and maintaining an additional set of PCs has a negligible cost. The Kelsa+ intervention involves a set of basic components, with the major costs being the upfront capital expense for the devices and software (operating system, internet browser, office productivity suite, anti-virus application). Additionally, there are monthly connectivity charges. There exist several procurement and connectivity options that determine the final cost of a given Kelsa+ facility. For a Kelsa+ facility with 40 workers, with capital costs amortized over a 3 year period, the monthly cost can range from $3.4 per worker (when a new PC is purchased, Microsoft Office is installed, and a 512 Kbps Unlimited Data Transfer connection is used) to $1.3 per worker (when a refurbished PC with Open Office is used, over a 256 Kbps Unlimited Data Transfer connection) [25]. (4) Development through ‘agency’ enhancement A key aspect of the Kelsa+ intervention is the freedom that accompanies exploration on the PC. The opportunity cost for workers to invest in their own learning is minimized through free access and convenient placement of the PC within their workplace. More importantly, by refraining from any restrictions on workers’ access to particular applications versus others, the agency of the worker is respected and encouraged in determining particular usages of the PC that meet his/her need [27]. Kelsa+ encourages a peer-learning model, through which workers with heterogeneous exposures and skills are allowed an opportunity to share their knowledge at a location where they congregate organically, i.e., the work place. Learning is driven by the user’s demands, and when not met by peers, workers are able to consult employees with more knowledge or step up and draw from training courses offered externally. In all this, the worker is centre-stage and his/her decisions dictate usage and impact, which differs from more paternalistic interventions in the ICTD space where certain ‘developmental’ results are expected and deviations from those results are treated as shortcomings. VIII LIMITATIONS AND ONGOING WORK Though based on a simple premise, Kelsa+ is a powerful idea because it takes the ICTD discussion and integrates it with the daily workings of mainstream institutions in developing countries. Many populations in need operate in or


close to existing IT infrastructures. As Kelsa+ shows, the innovations needed to have technology be useful to such groups are, therefore, mostly of political will, design, and process. There are many shortcomings in the current study. The evidence presented is based on a handful of cases, and such anecdotal positive effects cannot be mistaken for large-scale systematic impact. It may well be that the workers whose results we have recorded are exceptionally motivated and do not form the norm in this segment. The facility where the pilot was implemented was in fact a small software development and research centre, which may have presented many unique conducive factors for Kelsa+ to take off, including caring staff, an environment of trust, ubiquitous IT presence, strong IT maintenance support, 24x7 access, and open policies regarding employee PC usage. The interview responses from the participants may also have been influenced by the peer relationship they share with the researchers, which would prevent the sharing of negative feedback. It is clear that objectively verifying the value of Kelsa+ as a general ICTD intervention involves exploring it in a variety of institutional contexts. This is our ongoing effort. In extending the scope of Kelsa+ to become a systematic tool for socioeconomic mobility among urban low-income office service staff, our current efforts involve emphasizing longitudinal tracking of workers to map welfare impact, deploying Kelsa+ in non-IT corporate facilities, introducing some structured learning components to test their effect on learning outcomes and worker welfare, introducing certification options to prove skill acquisition externally during job-search, offering supplementary income generation possibilities for simple mini-tasks completed at the Kelsa+ PC, understanding the specific constraints that prevent women workers from using the Kelsa+ facility, and testing motivational tools that might influence the usage of the PC for explicit skill-building across domains and workers. This study draws the ICTD community’s attention to a subset of the poor who spend the bulk of their time around sophisticated IT infrastructures, but so far, do not gain from such proximity. We have described a pilot implementation of the Kelsa+ project in an office facility with 35 service staff, among whom the average worker earns $100 a month and has studied till class 10 in the local language. Over a period of 18 months, we saw broad improvements in workers’ confidence, self-esteem, and basic digital literacy, while a few individuals experienced increases in second-language (English) proficiency and career opportunities. Verifying these results through larger-scale and wider-scoped Kelsa+ implementations and measurements is now necessary to build on the promise of this simple, cost-effective, yet powerful ICTD intervention.

ACKNOWLEDGMENT To the workers involved in this project, we owe our most sincere thanks, for allowing us to examine and learn from their experience. Many thanks to the facilities management and IT staff at the pilot location for their support. Thanks to Itamar Kimchi for his help with refining the log analysis results. REFERENCES
[1] [2] [3] NASSCOM, ‘Indian IT Sector Score 10-in-10,” February 2007, URL: http://www.nasscom.in/Nasscom/templates/NormalPage.aspx?id=50833 NIIT: www.niit.com R. Dangwal, S. Jha, S. Chatterjee, and S. Mitra, “A Model of How Children Acquire Computing Skills from Hole-in-the-Wall Computers in Public Places,” Information Technologies and International Development, 2(4), 2005, pp. 41–60. M. Warschauer, Technology and social inclusion: Rethinking the digital divide. Cambridge, MA: MIT Press, 2003. L. Linden, “Complement or Substitute? The Effect of Technology on Student Achievement in India,” unpublished manuscript, Columbia university, June 2008. URL:http://www.columbia.edu/~ll2240/Gyan_Shala_CAL_2008-0603.pdf . Last accessed September 22, 2008. A. Lane, and M. Porch. “Computer Aided Learning and its impact on the performance of non-specialist accounting undergraduates,” Accounting Education, 11 (3), 2002, pp. 217-233. The One Laptop Per Child (OLPC) initiative: www.laptop.org ; Multipoint for education: http://www.microsoft.com/unlimitedpotential/programs/multipoint.mspx T. Oppenheimer, The Flickering Mind: Saving education from the false promise of technology. Random House Publishers, 2003. A. Mukherjee, “Build Robots Create Science – A Constructivist Education Initiative for Indian Schools”, Proceedings of Development by Design (dyd 02), Bangalore, India, 2002. R. Wang, et al. “Distance Learning Technologies for Basic Education in Disadvantaged Areas” Proceedings of the 8th Global Chinese Conference on Computers in Education (GCCCE 04), 2004. W. Schinagl, “New learning of adults in information and knowledge society”, Journal of Universal Computer Science, 7 (7), 2001, pp. 623628. URL:http://www.jucs.org/jucs_7_7/new_learning_of_adults/Schinagl_ W.pdf R. Heeks, “ICT4D 2.0: The Next Phase of Applying ICT for International Development,” IEEE Computer magazine, June 2008, pp. 26-33. S. Bailur, “The Complexities of Community Participation in Rural Information Systems Projects: The Case of ‘Our Voices’”, Proceedings of the 9th International Conference on Social Implications of Computers in Developing Countries, São Paulo, Brazil, May 2007. K. Kiri, and D. Menon, "For Profit Rural Kiosks in India: Achievements and Challenges," i4d magazine, June 2006. URL: http://www.i4donline.net/articles/currentarticle.asp?articleid=700&typ=Features R. Veeraraghavan, N. Yasodhar, and K. Toyama, “Warana Unwired: Replacing PCs with Mobile Phones in a Rural Sugarcane Cooperative,” Proceedings of the 2nd IEEE/ACM International Conference on Information and Communication Technologies and Development, December 15-16, 2007, Bangalore, India, pp. 89-98. R. Kuriyan, I. Ray, and K. Toyama, "Integrating Social Development and Financial Sustainability: The Challenges of Rural Kiosks in Kerala," Proceedings of the 1st IEEE/ACM International Conference on Information and Communication Technologies and Development, May 2006, Berkeley, USA. N. Rangaswamy, “Social Entrepreneurship as Critical Agency: A study of Rural Internet kiosks,” Proceedings of the 1st IEEE/ACM International Conference on Information and Communication Technologies and Development, May 2006, Berkeley, USA.

[4] [5]

[6] [7] [8] [9] [10] [11]

[12] [13]





[18] J. Pal, “Examining e-literacy Using Telecenters as Public Spending: The Case of Akshaya,” Proceedings of the 2nd IEEE/ACM International Conference on Information and Communication Technologies and Development, December 15-16, 2007, Bangalore, India, pp. 59-67. [19] A. L. Ratan, and S. Satpathy, “Digital Habituation as a Basis for Digital Inclusion,” Microsoft Research Technical Report, June 2008. [20] U.S. Pawar, A.L. Ratan and S. Blagsvedt, “An ‘Office Hole-in-the-Wall’ Exploration,” Microsoft Research Technical Report, January 2008. [21] A.L. Ratan, “Lessons from Low-income Workers in Bangalore on the Value of Information Technology.” Paper presented at the Conference on Living the Information Society: The Impact of ICT on People, Work, and Communities in Asia, Manila, Philippines, April 23-24, 2007. [22] R. Veeraraghavan, G. Singh, K. Toyama, and D. Menon. “Kiosk Usage Measurement using a Software Logging Tool,” Proceedings of the 1st IEEE/ACM International Conference on Information and Communication Technologies and Development, May 2006, Berkeley, USA, pp. 317-324. [23] The Rosenberg Self-Esteem Index: http://chipts.ucla.edu/assessment/Assessment_Instruments/Assessment_f iles_new/assess_rse.htm [24] Key English Test (KET) certification preparation, Cambridge University. URL: http://www.cambridgeesol.org/exams/generalenglish/ket.html [25] L. Zia, “Internship report: Digital Literacy for Low-Income Workers”, unpublished, Cambridge University, 2008. [26] R. De, and A.L. Ratan, “'Whose Gain is it Anyway?' Structurational perspectives on deploying ICTs for development in India's microfinance sector,” Information Technology for Development, forthcoming. [27] A. L. Ratan, and S. Bailur, “Welfare, Agency and ‘ICT for Development’,” Proceedings of the 2nd IEEE/ACM International Conference on Information and Communication Technologies and Development, December 15-16, 2007, Bangalore, India, pp. 119-130.

Figure 6: Distribution of applications used at the Kelsa+ PC over 18 months. Dominance of internet usage can be clearly seen. Decline in use of applications like offline games also seen.


Figure 7: Distribution of internet usage at the Kelsa+ PC per fortnight over 18 months. Increase in Internet video usage is clear. Steady and high usage of email is seen after fortnight 15. Social networking websites see increased usage starting in fortnight 21.


Mapping the Dynamics of Social Enterprises and ICTD in Cambodia
Kelly Hutchinson, Alemayehu Molla
opportunities. For these groups, social enterprises can be an important source of jobs, income, training, business mentoring, and technical input [2], [3], [35]. As ICTs and social enterprises become drivers of economic growth, it is important to investigate this nexus to see what opportunities these new models of business can bring to communities in developing countries. The study takes ebusiness as a typical instance of ICTs and argues that social enterprises are well positioned to benefit from the application and use of ICTs and to deliver the promise of e-business benefits for development. The paper explores how social enterprises are using ebusiness for their comparative advantage, and the outcomes (benefits) of e-business to social enterprises and the impact (attributable to e-business) of social enterprises in the target community. The study is situated in Cambodia where there are many social enterprises active in IT services, handicraft, hospitality and processing/production., Cambodia, therefore provides a space where a vibrant social entrepreneurial spirit presented a unique perspective on the perception, use and benefit of ICT in an international development context. Through mapping organisations in the Cambodian social enterprise sector, this paper aims to fill the knowledge gap, i.e., empirical evidence on the impact of ITCD, by reviewing donors’, associations’ and NGOs’ experience in order for others to learn how to use ICT to benefit their communities. Exploring how ICT is enacted in practice by social enterprises, although descriptive in nature, represents an original contribution to the body of knowledge and practice of ICTD. II. LITERATURE REVIEW ICT was promoted as offering businesses in developing countries a potential for creating new exchange mechanisms to enable them to compete on a more equal basis in world markets [38]-[41], [37], [45]-[47]. Apart from donor and multilateral agencies, there is a growing body of academic research such as SMEs’ uptake of ICTs in Botswana [12], the contextual study into ICTs in international development agenda in Vietnam [4] and the issues facing sustainability of community ICTs [15]. Following on from this nexus between ICTs and development is more applied research which builds on the agenda of donor agencies regarding the uptake of ICTs by SMEs and the role of change agents in facilitating this process [13].

Abstract— As Information and Communication Technology
(ICT) and social enterprises become drivers of economic growth, the nexus provides opportunities for new models of business to bring benefits to communities in developing countries. Recognising the complex dynamics and range of actors in this diverse and emerging sector, this study chooses to document the external influences, use and impact of ICT on social enterprises (SEs) -enterprises that have both a business and social development goal. The problem investigated is the potential gap between the rhetoric of the ‘promise of e-business’ versus the dynamics of enactment and impacts of ICT in practice in the social enterprise sector. The main contribution of the research is to identify the real development impact of ICT use by social enterprises by assessing the role of ICTs in achieving their goals. Its major finding is that SEs’ social and business missions are inextricably linked .and their use of ICTs and benefits from it can be used as a proxy to assess ICTD. Thus SEs provide a new paradigm in the study of ICTD. Index Terms—ICTD, ICT, social enterprise, international development, community informatics, Cambodia, CICs.

he usefulness of ICTs to achieve development goals are increasingly accepted by international development agencies such as the United Nations (UN) and the World Bank. As ICTs are relatively new tools within the development arena, their impact is emerging through the work of a number of researchers within the new discipline, ICT for Development (ICTD). Whilst the position of most of the multilateral agencies is grossly optimistic about the potential of ICT for addressing development needs of enterprises in developing countries, there is a need for empirical evidence as to the real impact of ICTs such as e-business in developing countries This paper focuses on the intersection of social enterprises (SEs) as dual social-business mission organisations, and the application of ICTs by those organisations in an international development context. Social enterprises generate revenue to support their social mission, as well as providing employment for often marginalised people [8], [9]. Social enterprises do business with an explicit social mission–helping poor and marginalized people to participate in sustainable business
Manuscript submitted September 22, 2008.




Whilst the position of most of the multilateral agencies is grossly optimistic about the potential of ICT for addressing development needs of enterprises in developing countries in general and that of SMEs in particular, there is an acknowledgment that tailored initiatives (such as e-commerce) are best suited for ICT to achieve development goals [20][21]. However, E-commerce has found it challenging to deliver the purported benefits espoused in the early studies of UNCTAD [38] and the OECD [30]. For instance, the DFID Globalisation and Poverty study explored the benefits of ebusiness for developing countries found that such benefits are largely missing and can only be realised if developing countries’ enterprises build their ability to develop tailored marketplaces [22] , [31]. More specific research [27]-[28] into the reality of e-commerce benefits among businesses in developing countries also cautioned the over-optimistic expectations of e-commerce benefits. Overall, current evidence appears to suggest that for e-commerce to become more widespread in a way that benefits producer firms in developing countries, greater attention will need to be given to how firms relate to each other within global value chains [25], as well as the specific types of transactions they are involved in. Core to this debate is the call for empirical evidence as to the real impact of ICTs such as e-business in developing countries– ‘whose development does e-commerce effect?–as raised by [30]. Finally the call for ‘bottom-up’ approaches in developing countries that are based on realistic assessments of ICTs and e-business opportunities and obstacles, and region and value chain specific solutions provide impetus for this current research. This study argues that social enterprises are one form of enterprise that are likely to benefit from the application and use of ICT and can then deliver the promise of e-business benefits for development. Emerson & Twersky defined a social enterprise as a ‘generic term for a non-profit enterprise, social-purpose business or revenue-generating venture founded to support or create economic opportunities for poor and disadvantaged populations while simultaneously operating with reference to the financial bottom line’ [14]. Davis [8] sees social entrepreneurship as extending the definition of entrepreneurship by its emphasis on ethical integrity and maximizing social value rather than private value or profit. The diversity of the emerging field prompted Alter to write the Social Enterprises Typology [1]. Alter’s definition captures purpose, approach and practical applications: A social enterprise is any non-profit-owned revenue-generating venture created for the purpose of contributing to a social cause while operating with the discipline, innovation and determination of a for-profit business [1]. Key to the understanding of social enterprises is their focus on sustainability and their dual social and business mission. Sustainability is a core principle of social enterprises and refers to a business venture’s simultaneous pursuit of beneficial outcomes along three dimensions: economic, social and environmental. As a way forward in building sustainable

communities, social enterprises generate revenue to support their social mission, as well as providing employment for often marginalised people [8], [9]. A social mission is the clearly stated goal of providing benefit to some section of society. This requires identifying the target group the enterprise wishes to support. It also requires clear statements of intent to make sure that it is understood that the purpose of the organisation is to meet the social, as well as the business mission. The growing relevance of social entrepreneurship for development builds on the success of SMEs in reducing poverty and improving livelihoods The intersection of ICT and social entrepreneurship therefore provides a way forward in building sustainable communities. As a framework Alter’s [1] social enterprise typology illustrates how social enterprises have come up with ownership models, income and capitalisation strategies, and the unique management and service systems designed to maximize social value. The international development focus has shifted in recent years to building sustainable enterprises and encouraging the emergent SME sector [10], [42]. In acknowledging this shift and increased focus on economic and social sustainability we find that social enterprises are playing an increasing role with the support of donors, governments and communities. III. ASSESSMENT FRAMEWORKS In order to assess the outcome and impact that ICT may deliver to social enterprises this study considers how social enterprises’ use ICT to achieve their dual mission and how they in turn measure success. When considering how to measure impact, the line of investigation reviewed a range of areas including livelihood and capability frameworks, and current work in the field of ICTD. Impact can be measured using livelihood and capability frameworks [28]. In particular DFID’s Sustainable Livelihoods Framework (SLF) is an effective tool for uncovering and highlighting the relationships between the livelihoods of vulnerable people, businesses and the institutional context in which business transactions take place. The challenges facing the social enterprise sector are similar to those of other non-profit organisations where the social mission has driven programs yet there has been little thought how to measure the achievement of their social goals. Some effort has been put into developing tools to assess the impact of social enterprises in what is commonly known as ‘social accounting’. Payne clearly states that social accounting is not a social impact assessment as the official title implies, ‘Social and Ethical Accounting, Auditing and Reporting’ (SEAAR) [33]. Built around stakeholders, AA1000 (http://www.accountability.org.uk) seeks to link the defining and embedding of an organisation's values to the development of performance targets, thus tying social and ethical issues into the organisation's strategic management’ [33, p14]. This research assumes that social enterprise impact is predicated on the organisation's mission, the social objectives it intends to


achieve, and what impacts can be measured. Figure 1 based on Alter [1] provides an overview of the research framework and Table 1 details some of the key concepts to be explored.

research [6], [22], [24]. A case study may capture reality in greater detail, analysing a greater number of variables than is possible with other research methods [15]. Yin [47] recommended the use of case-study protocol using a triangulation of data sources. This study looks at the key actors in the Cambodian social enterprise environment in an effort to answer the research question. As this study addresses a new area of academic inquiry there was little primary data regarding the topic, it requires a range of research methods and active dialogue with organisations involved in the sector. Triangulation for this study was created through interviews, surveys and document analysis.

Fig. 1. Research Map

KEY CONSTRUCTS OF INTEREST Construct Beneficiaries Definitions Target community who benefit from SEs activities Ways in which social enterprises provide support, Examples Disabled, Women, Street Children, Orphans, People Living With HIV/AIDS, Youth, Unemployed, Trafficking Survivors, Villagers, Farmers Employment, Training/ Skills Development, As Consumer To Local Producers (Creating Market), As Marketer Of Local Producer’s Products (Intermediary), Supplier to Local Producers, n, Assistance In Business Development, Product Design/ Service Development An organisation conducts a social audit makes itself accountable to its stakeholders and commits itself to following the audit's recommendations. Evaluate Performance against Social Indicators, Ability to Employ More People, Recommendations from Customers, Recommendations from Beneficiaries, Awards/ External Recognition from Peers, Specify, Awards/ Recognition from Development Agencies/Donors Sub-sectors Handicraft Associations Artisans Association of Cambodia Craft Network Cambodian Silk Forum ICT:CAM Social Enterprises Hagar Design Handmade Textile Association Colours of Cambodia Shinta Mani CIST Community Info Centres Digital Divide Data Yejj Info Senteurs D’Angkor


Hospitality IT Services

Forms of Support

Processing Multi-sector WASMB

Social accounting and social auditing7

Methods of measuring and reporting an organisational social and ethical performance. A set of measures of determining if the social mission is being met

Social Mission Indicators

Participants were both industry protagonists and those that support them through associations and donor agencies. Supplementary data was collected from a review of documents and artefacts such as brochures, marketing material and websites. Donor and media reports were sought regarding the activities of these social enterprises in order to ensure as complete a picture of the social enterprise sector as possible. Of the 50 organisations identified in the study, 18 were interviewed and 25 were surveyed. 1) Defining Beneficiaries The next step is to identify the target beneficiaries of social enterprises’ blended mission. A useful tool from the UK is the DETR 2000 Index of Deprivation – a comprehensive index for comparing levels of deficiency across a range of issues. The index covers seven aspects of deprivation – income, employment, poor health and disability, education skills and training, housing, geographical access to services and child poverty. The framework was used in this study to assist interviewees in identifying target beneficiaries. The resultant categories were then used in the survey and are presented in Figure 3. below.

IV. METHODOLOGY In order to bridge the gap between theory and practice a qualitative approach was chosen as applied research leans towards involving practitioners, users and other stakeholders, in order to provide practical outcomes. The research took a holistic–multi-disciplinary, multi-method–approach to triangulate data collection and is detailed as to sample and selection, cross-sectional survey, semi-structured interviews, information observation, document analysis and artefact review as utilised in the case study. Examining a case of one sector’s relationship to ICT and development allows an exploration of particularity and complexity of a single case to understand its activity within important circumstances (Stake 2000) utilising the case study method of exploratory IS


Fig. 3. Target Beneficiaries

The categories in descending order represent lesser degrees of importance. The findings showed a high incidence of support for beneficiaries with disabilities; however women and youth are also a major focus of social enterprises in Cambodia. As this data is aggregated across different sub-sectors, another picture emerges. The handicraft sector supports beneficiaries from all categories, including street children and people living with HIV/AIDS, which neither IT services nor social enterprises from production sub-sectors supported. Although the survey provides just one set of data, the interviews and document analysis also identified that some other social enterprises supported these disadvantaged people in Cambodia. Mith Samlanh works with street children through their restaurants Friends and Romdeng, and Khana is an association working with NGOs in support of people living with HIV/AIDS. Both recognize the potential that income generation activities can have in creating sustainable livelihoods for their target beneficiaries. Now we turn to social enterprises themselves to measure their impact on their target community and see how ICT does or does not play a role in achieving their blended mission. The analysis will address the blended value proposition in conjunction with the social enterprise typology to determine the outcomes and impact of ICT on social enterprises. V. KEY FINDINGS Outcome and impact of ICT – achieving social enterprises’ mission The paper now turns to assess if ICT does deliver benefit to social enterprises by considering how ICT helps them to achieve their blended mission which combines commercial and community goals. It is hoped that the evidence presented here will contribute much needed insight into the impact of ICTs on development more generally. A. Blending the dual mission In addressing how the dual mission of social enterprises is enacted in practice this paper asks in turn, how ICT interacts in this context. The new organisational model of social enterprises embodies a shift in the pursuit of sustainable development on all levels, financial, social and environmental. This move to a ‘blended value proposition’ [14] reflects the integration of the dual mission, which provides the context as

to the contention of this paper – that it is due to the very nature of social enterprises that ICT does bring benefit directly as it assists the overall mission and the ‘total value creation’ that social enterprises bring their beneficiaries. Initially it was thought that the benefit that social enterprises bring communities could be divided into two sides of the dual mission, business and social as illustrated in Fig. 1. However, as the research process unfolded, the realisation that the blended mission drives social enterprises had a profound influence on the resulting framework for analysing how ICT delivers benefit to target communities. To understand how this is manifested in practice, Table 3 provides some examples of organisations’ statements of their mission from the survey results.
TABLE 3 EXAMPLES OF SOCIAL ENTERPRISE MISSIONS CIST Colours of Cambodia Hagar Design Khmer Mekong Food Rajana Association Senteurs D’Angkor SotheaKhmer Silver Crafts Bringing digital opportunities to underprivileged young people, that is developing an IT training centre Provide fair trade channel for small producers and NGO products Our social mission is to train and employ highly disadvantaged Cambodian women while using our profits to financially support Hagar. Great jobs for young people and students Buying raw agriculture from farmer to push them grow more fruit and vegetable Provide nutrition and quality Safe food to consumers Find Markets for small producers, give them the employment, send them to school Good working conditions, share benefits with staff through a bonus system. Create employment for local villagers Social welfare for people with disability and vulnerable poor people. Work and wage for poverty

What is of interest is the intertwining of both business and social goals in many of the respondents answers. This suggests that the mission of these organisations is a blended proposition providing evidence of this new ‘business’ segment operating with specific commercial and community development goals. 1) Reframing to reflect the blended mission As a result of this investigation, there has been a slight departure from the originally perceived dual mission to the more blended version which is reflected in Figure 2. the Social Enterprise Blended Mission, which highlights the potential outcomes of ICT in social enterprises. As this is a qualitative study, following naturalistic inquiry, it is usual for the research questions to evolve over the period of investigation. What was the original understanding, that the dual mission was two separate yet complementary missions, was due to the perspective as an outsider. Through examining the data it became evident that this distinction is more blurred than originally expected.

INDICATORS OF IMPROVED LIVELIHOODS Indicator Earning a livable wage Provided new career opportunities Improved food security Fig. 2. Social Enterprise Blended Mission Improved family stability Improved health Improved access to education Increased purchasing power % agreement 73% 63% 71% 71% 57% 57% 71% TABLE 6

As Figure 2. demonstrates, it is difficult to differentiate between the business and social drivers within the blended value proposition. Improved communication can be both with donors regarding monitoring of a social program such as DDD and UNDP, or to customers purchasing silk handbags for resale in Australia, in the case of Hagar Design through Craft Network. The findings indicate that there is not a clear distinction between an NGO that earns income from selling craft, such as Tabitha Cambodia, to an IT company which has a policy of employing disadvantaged young people, such as Yejj Info. In fact, Yejj Info provides one of the best examples of the blended value proposition with their comprehensive mission statements which combines commercially, socially, ethically and environmentally sustainable 2) Mapping the blended mission in action Social enterprise impact is predicated on the organisation's mission, the social objectives it intends to achieve, and what impacts can be measured as a result of the business initiatives. A defining ambition of social enterprises is that they support those disadvantaged in the community. Leading by example, social enterprises employ disabled computer operators at DDD, market landmine survivor’s art work at Colours of Cambodia, and encourage local communities to invest for their future through a community savings program at Tabitha Cambodia. Exactly how this complex interweaving of social and financial sustainability is achieved is what this section aims to address. This manifestation of dual sustainability illustrates the true outcomes of the blended mission. What this study acknowledges is that these are inextricably linked, so to measure one in isolation of the other denies the new paradigm that social enterprises present. B. Measuring the social impact Social impact indicates a positive effect on the target population as a result of an intervention and can be measured. Social enterprises, like all development programs, have direct as well as indirect impacts. Alter [1] provides some examples of social enterprise impact measurements and corresponding indicators. These were adapted to identify the outcomes (benefits) of ICT to social enterprises and in particular to assess the impact (attributable to ICT) of social enterprises in the target community. Survey respondents were asked ‘In your opinion how has employment with your organisation improved the livelihoods of your staff?’ and their response is shown as percentages of those who either strongly agreed or agreed with the indicators listed in Table 6 below.

The nature of social development lends itself to monitoring and evaluation [11], and often the indicators for these outcomes are clearly set during the start up of the venture. In the case of donor initiated enterprises there is often a logical framework or ‘logframe’ that sets out indicators for monitoring and evaluation. In the case of social enterprises the need to measure the triple bottom line means a range of indicators from both social and business areas are required. In evaluating the impact of the current sample, a number of indicators were presented to social enterprises in the survey. The 44% incidence of evaluating performance against internally structured social indicators is promising as it suggests that these organisations are very clear of the need to meet their dual mission. This potentially could mean they will be more successful than an organisation that is not as clear on their strategic mission. Another category for measuring the success of achieving the social mission is through recognition from stakeholders. This included acknowledgement from peers, awards, and notes of appreciation from the most important people, beneficiaries and customers which accounted for the highest response rate. Although the survey data had a low incidence of recognition from development agencies/donors through awards, the other data collection discovered a number of awards and pride in showcasing social enterprises success in Cambodia is further explored below. C. Identified benefits of ICT It is through the case of the Cambodian social enterprise sector that this paper demonstrates the opportunity to address social issues through business initiatives. Benefits of using ICT, either directly as in the case of IT services with skilled labour, or indirectly by selling via an ICT-enabled intermediary. If one of the promised benefits of ICT is increased efficiency of both internal and external operations then it is important to measure how ICT impacts on social enterprise operations. In measuring the extent of ICT utilisation in the organisations surveyed, the perception on impact on operations was deemed an appropriate measure. The most common way ICT helped improve operations was through enabling external communications with clients and customers. This was further supported by the interview results, with most social enterprises seeing ICT as a core tool for customer relationship management. Simple ICTs such as phone and email were the dominant forms.



Access to Markets

Does ICT enable social enterprises to link into the global trade of goods and services? Through the application of ICT are social enterprises able to access new markets that they previously would have been excluded from? The fair trade movement is a response by civil society to the potentially negative impact of globalisation [33]. It is important to acknowledge that the networked economy can potentially further marginalise producers in developing countries. Those wishing to redress this imbalance see the benefits that ICT can bring as a tool for trade [5]. The majority of enterprises surveyed felt that ICT provided benefit in key areas of communication with customers (80%), and sales and marketing (64%), suggesting that social enterprises are using ICT to connect to markets and facilitate trade in ways previously not imagined.
b) Income Generation

when direct sales are for goods produced by the target beneficiaries, customers like to know their purchase actually goes to help those who created them. This is where ICT can assist with clear communication and monitoring. An example of how this is enacted in practice can be seen on the Khmer Silk Processing Association (KSPA) website (See http://www.phaly-craft.org) where they state clearly the promise that the income goes to those who need it. All income from the sale of these products are partly used to support Future Light Orphanage (FLO) activities and children whose numbers increase everyday.
c) Employment Opportunities

One of the defining characteristics of a social enterprise is its income generation activities. This is in fact what differentiates them from others in the non-profit sector as the goal is clearly to earn income to support the social and business development of the organisation. What is of interest to this study is the role that ICTs have played in achieving this goal. Respondents to the survey were asked to respond to a number of statements which variously reflected their perception of the impact of ICTs on achieving increased sales or income. These statements were developed from the pilot assessment of DDD and some preliminary interviews with other social enterprises to identify the various ways impact is measured. The level of agreement is a combination of agreed and strongly agreed and suggests how these social enterprises see the relationship between income generation, employment and ICTs.
TABLE 7 ICT ROLE IN ACHIEVING GOALS How do these statements apply to your organisation? Email has increased our sales ICT has increased our profits We have employed more staff as ICT makes it easier for us to do more work for our clients Due to increased sales from using ICTs (website/ email) we have employed more staff Due to increased income from ICT related we have been able to offer more social programs to staff % agreement 50% 33% 20% 40% 40%

Choosing an appropriate target group requires prior knowledge and theoretical understanding of the social environment of the specific country. DDD identified their target group as people who fit one of these criteria: disabled, orphans (due to war or HIV/AIDS), rural poor and trafficking survivors. They also focus on youth providing entry-level positions and training in order to develop their career opportunities. DDD found that it is important when dealing with disabled people to give them the opportunity to prove their skills. On first appearance a young woman with no hands below her wrist may seem to have limited potential as a data entry operator; however once given the chance she has become a valued employee who upholds the required productivity levels without having any fingers. This is a prime example of how DDD has taken on the challenge of providing employment opportunities for disabled people and met both individual staff goals and general business needs.
d) Capacity Building

As the blended value proposition of the social enterprise is an interweaving of social and business indicators, it is not surprising that ICT helps to achieve both these simultaneously. Hence separating the business versus social is not that useful, for it is this unique blended characteristic that perhaps makes the impact of ICT more successful than if it was separated into a silo approach for development. One concern with income generation is that the proceeds go to support the beneficiaries and are not inappropriately used for administration or unrelated costs. Of course this is entirely dependent on the structure of the social enterprise; however

Some social enterprises’ social mission encompasses capacity building, where target beneficiaries are provided training and work experience in using ICTs directly. In this case the social mission drives the adoption of ICTs. In the hospitality industry students are trained on hotel booking systems such as at Shinta Mani and are able to be prepared for real life experience not just the theory that class based learning alone provides. By embracing the experiential learning approach, ICTs can be used to simulate real life experiences and provide an advantage for skill development. One of the major concerns of the digital divide is not only physical connectivity the ‘second term level digital divide’ [18]. Skills to effectively use ICTs highlight that connectivity alone is not enough to bridge the gap. It is with this in mind that some social enterprises, particularly in the IT services sector, are providing direct benefit to their target communities. DDD, Yejj Info and CIST all have training and skills development as a core program of assistance. Another social enterprise in the IT sub-sector, Yejj Technology, operates a Cisco Network Academy which provides formal training programs with industry recognised qualifications so that their students from disadvantaged backgrounds can engage in the IT services sector.



Strengthening Sponsor Relations

One unique benefit of sponsorship in the highly competitive non-profit sector is that a personal relationship is built between a sponsor and a community. It is this connection that keeps them returning or undertaking fundraising activities in their home country continues sponsorship of either an individual student as in the case of Shinta Mani, or new facilities for the Future Light Orphanage, which is run from the proceeds from the KSPA. Communication via email is a key tool for this, enabling all phases of the relationship, prior to visit and post visit, to continue seamlessly. KSPA has also branched out to provide some guest accommodation at the orphanage on the outskirts of Phnom Penh. Enabling those who wish to make their contribution to the NGO through not only donations, or purchasing goods or services, guests can volunteer to work with the children teaching English or computers providing a true link to the community. The opportunity to personally interact with the children is actively marketed on the KSPA website and also has an online booking form to ease the process. The form is then submitted and emailed to KSPA and then they communicate with the customer via email and finalise the booking and program for their visit. It is the seamless ability to engage and interact with KSPA even via a simple online form that shows how ICT facilitates this relationship. Cambodia unfortunately attracts paedophiles who visit Cambodia to take advantage of poor and disadvantaged children and women. The management of FLO take the care of their children very seriously and screen visitors to ensure they are not placed at risk but benefit from interacting with only genuinely interested volunteers who match KSPA's concern for the welfare of the children as a priority.
f) Social Outsourcing

Focusing on the philanthropic benefit of outsourcing to disadvantaged communities has been a unique strategy to counteract the backlash that US firms face as they outsource jobs overseas. Maximising this market niche, DDD shows that business can benefit communities in developing countries in a viable business model that does not exploit workers. DDD in fact targets beneficiaries from marginalised backgrounds and provides employment where others reject them. Through their social mission they provide educational scholarships that support their workers career development to leave DDD and take up new job opportunities. This then provides an opening for other disadvantaged Cambodians to join the workforce. Focusing on the staff’s professional development and achieving the business goal provides much more of an impact for local communities than charity. Businesses who value social contribution are happy to partner with social enterprises such as DDD as they can satisfy their needs to reduce costs through accessing an ethical supply chain. All of which is enabled by ICT as digitising content is the mainstay of DDD’s services. DDD also understand the benefit of social marketing and the role it can play in bringing a comparative advantage to

competitors. They have actively sought media coverage and see it as a way to promote the ideals of their social enterprise to their potential market. With the backlash about outsourcing jobs overseas a public awareness campaign in the US is very important for an enterprise such as DDD. On their website they have a News section which lists articles and stories which highlight the experience of the organisation from an external perspective. As the article on their website illustrates, DDD knows the benefit their business brings young Cambodians like Nut Pove, the opportunity they would otherwise have missed. But it is sadly not enough to provide jobs alone, this is where ICT can help spread the word of the work being done in Cambodia and it is examples such as DDD which provide evidence of how this is being done. Although social outsourcing is more commonly connected with the IT services sector, the model is also used in the manufacturing sector. In the case of handicraft sector in Cambodia, many social enterprises work with designers from international markets to produce their products. Small boutique labels make connection with suppliers in Cambodia through agents such as AAC and Craft Network to arrange production. For example the designer in the US or Australia send design, colour specifications via email and the production is made to order. As Cambodian enterprises can handle small quantities and offer individual attention to design houses they are increasingly popular. Bronwyn Blue a Small Business Development Consultant, specialising in the craft sector, works with social enterprises to improve their quality control and designs to meet the needs of international market: The core finding presented in this paper is how the blended mission is enacted in practice and in turn, how ICT interacts in this context. This study argues that it is the integration of the dual mission, which is the very nature of social enterprises, that ICT brings benefit directly as it assists the overall blended mission and the ‘total value creation’ that social enterprises bring their beneficiaries.

Fig. 4. Map Total Value Creation



This study found that the major characteristics of social enterprises are that their development objectives are to achieve a blended mission meeting both social and economic goals. From this basis the assumption would be that ICTs should be utilised to realize these goals [19], [7]. It is with this


in mind that this study aimed to find a realistic view of the role ICTs play in supporting social enterprises to meet their dual mission. What this study acknowledges is that these are inextricably linked, so to measure one in isolation of the other denies the new paradigm that social enterprises present. Social enterprise impact is predicated on the organisation's mission, the social objectives it intends to achieve, and what impacts can be measured. Social enterprises, like all social programs, have direct as well as indirect impacts. Alter [1] provided some examples of social enterprise impact measurements and corresponding indicators. These were adapted to identify the outcomes (benefits) of ICT to social enterprises and in particular to assess the impact of social enterprises in the target community. In evaluating the impact a number of indicators were presented to social enterprises in the survey and the results indicated that evaluation of performance was against internally structured social indicators such as improved livelihoods, increased income, access to affordable health care and access to education. This is promising as it suggests that these organisations are very clear of the need to meet their dual mission. This potentially could mean social enterprises will be more successful than an organisation that is not as clear on their strategic mission. The goal for development of disadvantaged communities can take many forms and in the case of social enterprises there appears to be a number of models as outlined in Alters Social
DDD Hagar Design Yejj Info Shinta Mani KSPA Tabitha

establishing the social enterprise and of those surveyed three were still registered as an NGO whilst eight others were set up as social enterprises from the outset. The summary of results in Figure 8.e shows a number of social enterprises as classified according to the SE Typology. The level of ICT uptake is not directly linked to the level of integration however it could be claimed that there is some correlation with full integration and management vision with more successful ICT assimilation. This study argues that it is the integration of the dual mission, which is the very nature of social enterprises, where ICT brings benefit directly as it assists the overall mission and the ‘total value creation’ that social enterprises bring their beneficiaries. This is promising as it suggests that these organisations are very clear of the need to meet their dual mission. This potentially could mean social enterprises will be more successful than an organisation that is not as clear on their strategic mission. This bears direct significance on the framing and uptake of ICTs. The institutional form that the social enterprise takes itself is another potential influence as to the uptake of ICT. Management awareness, capacity of staff and commitment to use ICT to achieve the blended mission requires both sides to be successful where meeting economic needs is equally as important and social goals.

Enterprises Typology [1]. If we consider the social enterprises sector as a norm due to its accepted social constructs of philanthropy and social good, then this in itself could act as an influence on the enterprises uptake of ICT. So it could be claimed that the type of social enterprises may also impact on the uptake of ICT. Perhaps it is the ways the social enterprise enact their social mission that predicates how they will or will not utilise ICTs as the following examples illustrate. The form that a social enterprise takes is an interesting entry point for analysis as represented in Figure 5. Using Alter’s social enterprise typology to analyse the survey data, regarding how social enterprise perceive themselves, can provide some insight into the level of maturity and development of the organisation. One of the approaches to assessing social enterprises is to classify them based on the level of integration between social programs and business activities and this framework helped develop the research design for this current study. This is where the social enterprise typology can assist in analysis of the experience of this sector in Cambodia. Interview results indicated that two social enterprises that were established by NGOs as income generating initiatives have made the transition to being an independent social enterprise. The question this raises is to what extent does the relationship with the founding NGO continue once the transition has occurred? Interview data suggests that NGOs are instrumental in

Fig.5. SE Typology - Business/Program Integration [1]

The final aim of the research was to measure the role of social enterprises’ use of ICTs in achieving their goals. This study’s major finding was that the blended mission requires both sides to be successful, thus meeting economic and social goals as equally as important. In identifying at what point ICT intervention made an impact, the areas that stand out include income generation, access to markets, improved communication with customers, and strengthening of sponsor relations. Whilst the results indicate a positive acceptance of ICT, most social enterprises are still at the early stages of adoption. This means that there is a continued role for external institutions to support further development, with a suggested focus on meeting the dual mission in a sustainable manner, and therefore meeting the requisite standards for capacity and use. Most importantly this study acknowledges that the social and business missions are inextricably linked, so to measure one in isolation of the other denies the new paradigm that social enterprises present. Perhaps the most active sub-sector that best reflects this unique value proposition is the emerging IT sector. Social outsourcing represents a new model and the potential of the IT services sub-sector came to light during this research. In summary the case of the social enterprise sector in Cambodia provided a rich tapestry of examples with a number


of organisations receiving external recognition of Cambodia’s social entrepreneurs. The outlook for the other sub-sectors handicraft and processing have yet to fully engage and use ICT to their maximum benefit. As such there is a need for further support to best harness the potential that ICT can bring developing countries. VII. RECOMMENDATIONS If the aim of social enterprises is to have a positive outcome and impact for their target beneficiaries, then they should be open to develop standards that ensure no-one is exploited. Responding to the call to address issues in the regulation of global supply chains, and the potential that ICT can play in the self-regulation required in ethical trade provides another perspective to this study [22]. Through adhering to the standards of fair or ethical trade, Cambodia can build on its reputation that workers are safe and purchases make a real difference. This is an issue for workers in the home-based handicraft business model and increasingly will also become an issue for the IT outsourcing sector. As such one recommendation is to develop a fair trade charter for the social enterprise sector in order to ensure that those who are meant to be benefiting from the enterprise are in fact receiving a real benefit. The development of fair trade monitoring along the lines of the ILO Better Factories (See http://www.betterfactories.org) program is a model that could address this. This is a unique initiative that uses ICT to assist the process of transparent reporting in the garment sector in Cambodia. Using an interactive website, this project enables buyers to log in from overseas to check the status of a particular factory to see whether it has met Cambodian labour standards. One issue is that this model is also seeking to make the transition from a donor initiated program to a selfsustaining model. In fact the most active representative body in the social enterprise sector is the Artisans Association of Cambodia, which already has a working relationship with Garment Manufacturing Association of Cambodia via the Government-Private Sector Forum, so it is feasible that some mutually beneficial arrangements can be made. As there are some correlations between the manufacturing sectors and the sub-sectors of the social enterprise sector, there might be a possibility for a new holistic model to monitor social outsourcing, using ICT as a key tool to promote transparency. Another issue that arose from this research is the lack of regulation of the social enterprise sector in Cambodia, which the above recommendation would address. In the case of the NGOs making handicrafts, they are officially registered as an NGO with the Ministry of Interior, which has no reference for income generation. As such, NGOs do not pay tax, and the revenues from sales overseas are not being reinvested back into the broader economic development of Cambodia. Registration with Ministry of Commerce is required for any entity selling goods or services that may require a certificate of origin for export. These are areas that donors and other institutions working with social enterprises can assist and that

ICT can play a role in facilitating a more streamlined process for all actors. Finally, an area that poses potential is to review the long term impact of ICT in social enterprises in developing countries in terms of its contribution to poverty reduction and improved livelihoods. This study raises the issue of measuring the impact of ICTD-SE and it is recommended that further research be conducted to refine the framework. This would enable better assessment of the inherent challenges and potential impact that the intersection of ICT, development and social enterprises presents producers and communities in developing countries. In conclusion this study contends that the foundations are there for social enterprises to progress along the continuum from simple ICT use to integrated ICT and e-business. This combined with the increased level of awareness of the benefits and the supportive donor environment augurs well for Cambodian social enterprises to engage in broader use and therefore bring benefits of ICT in their development and positive impact in the lives of the target beneficiaries. REFERENCES
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] Alter, S. K. 2004) Social Enterprise Typology [online]. Virtue Ventures LLC. 2004. Badiali, N. (2005) Tall Poppies - Enterprising the Social Economy. The Scottish Council of Voluntary Organisations (SCVO) and Kilmarnock College. Bornstein, D. (2004) How to Change the World: social entrepreneurs and the power of new ideas, Oxford University Press. Boyle, G. (2002) Putting Context into ICTs in International Development: An Institutional Networking Project in Vietnam, Journal of International Development, Vol.14, No.1, 101-112. Caserta, A. (2001) Creating a Fair Trade Partnership through a Fair Trade Global Communication System. Cash, J. I. & Lawrence, P. (Eds.) (1989) The Information Systems Research Challenge: Qualitative Research Methods, Boston, Harvard Business School Research Colloquium. Curtain, R. (2004) Information and Communications Technologies and Development: Help or Hindrance?, Curtain Consulting for AusAID. Davis, S. (2002) Social Entrepreneurship: Towards an Entrepreneurial Culture for Social and Economic Development. Youth Employment Summit. Dees, J. G. (2003) New Definitions of Social Entrepreneurship: Free Eye Exams and Wheelchair Drivers. Knowledge @ Wharton, Wharton School, University of Pensylvania. DFID (2005) Working with the Private Sector to Eliminate Poverty. DFID & the Private Sector. London. DiMaggio, P. (2001) Measuring the Impact of the Non-profit Sector on Society is Probably Impossible but Possibly Useful: A Sociological Perspective. Measuring the Impact of the Non-profit Sector. P. Flynn and V. A. Hodgkinson. New York, Plenum: pp.249-272. Duncombe, R. and R. Heeks (2001) Enterprise Development and Information and Communication Technologies in Developing Countries: Supporting “ICT-Flyers”. Manchester, UK, IDPM, University of Manchester. Duncombe, R. and A. Molla (2005) An Evidence Based Framework for Assessing the Role of Change Agents in the Promotion of E-Commerce for SMEs in Developing Countries. Connecting people and places: challenges and opportunities for development, Milton Keynes, UK, UK Development Studies Association. Emerson, J. (2003) Blended Value Proposition: Integrating Social and Financial Returns. California Management Review, 45. Emerson, J. and F. Twersky. (1996) New Social Entrepreneurs: The Success, Challenge and Lessons of Non-profit Enterprise Creation. San Francisco, Roberts Foundation.



[14] [15]

[16] Galliers, R. (Ed.) (1992) Information systems research : Issues, methods, and practical guidelines, Oxford, Blackwell Scientific Publications. [17] Gurstein, M. (2005) "Sustainability of Community ICTs and its Future." The Journal of Community Informatics Vol.1 (2). [18] Hargittai, E. (2002)"Second-level digital divide: differences in people's online skills." First Monday Vol.7 (4). [19] Harris, R. (2004) ICT for Poverty Alleviation. e-Primer. Kuala Lumpur, APDIP. UNDP [20] Harris, R. and D. Vogel (n.d.) E-Commerce for Community Based Tourism (Draft) Hong Kong, Harris and Associates. [21] Heeks, R. (2002) Failure, Success and Improvisation of Information Systems Projects in Developing Countries. Development Informatics Group, IDPM, University of Manchester. Working Paper 11. [22] Heeks, R. & Duncombe, R. (2003) Ethical Trade: Issues in the Regulation of Global Supply Chains. Centre on Regulation and Competition. University of Manchester. [23] Humphrey, J., M. Robin, et al. (2003) The Reality of E-commerce with Developing Countries. Globalisation and Poverty, DFID Media @LSE and IDS. [24] Kaplan, B., Truex, D. P., Wastell, D., Wood-Harper, A. T. & Degross, J. (Eds.) (2004) Information Systems Research: Relevant Theory and Informed Practice, Norwell, MA, Kluwer Academic Publishers. [25] Kaplinksy, R. and J. Readman (2001) Integrating SMEs in Global Value Chains: Towards Partnership for Development, UNIDO. [26] Klein, H. M. & Myers, M. D. (1999) A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems. MIS Qtly.Viewed: 10 November 2006. [27] Molla, A. (2004) The Impact of eReadiness on eCommerce Success in Developing Countries: Firm-Level Evidence. Development Informatics Working Paper. Viewed: 20 November 2006. [28] Molla, A. (2005) Institutions and ICTs in the Makerere university: Bridging digital divide or spreading donor dependence,. IFIP Conference. Abuja, Nigeria. [29] Murray, C. (2001) Livelihoods research: some conceptual and methodological issues. Development Studies Association Annual Conference-Panel on Livelihoods Frameworks and Poverty Analysis, University of Manchester. [30] Odedra-Straub, M. (2003) "E-commerce and development: Whose development?" Electronic Journal of Information Systems in Developing Countries Vol.11 (2): pp.1-5. [31] OECD (2004) Accelerating Pro-Poor Growth Through Support for Private Sector Development, OECD. [32] Paré, D. J. (2001) Does This Site Deliver? B2B E-commerce Services for Developing Countries. B2B E-commerce and Developing Countries. Media@LSE. London, The London School of Economics & Political Science. DFID funded project on e-commerce for developing countries.UK [33] Payne, J. E. (2002) E-Commerce Readiness for SMEs in Developing Countries: A Guide for Development Professionals. Washington, LearnLink. [34] Redfern, A. & Snedker, P. (2002) Creating Market Opportunities for Small Enterprises: Experiences of the Fair Trade Movement. SEED Working Paper 30. Geneva. [35] Spinali, L. and H. Mortimer (2001) A Scan of the Not-For-Profit Entrepreneurship: Status of the Field and Recommendations for Action, Kauffman Center of Entrepreneurial Leadership. [36] Stake, R. (2000)Case Studies. Handbook of Qualitative Research. N. K. Denzin and Y. S. Lincoln. Thousand Oaks, CA, Sage Publications. [37] UN (2003) Tools for Development Using Information and Communications Technology to Achieve the Millennium Development Goals. Working Paper, United Nations ICT Task Force. [38] UNCTAD (2001) The Digital Economy: Integrating the LDCs into the Digital Economy. E-Commerce and Development Report 2001. Brussels, UNCTAD, Electronic Commerce Branch. [39] UNCTAD (2002a) World Telecommunication Development Report: Reinventing Telecoms, UNCTAD. [40] UNCTAD (2002b) E-Commerce and Development Report 2002. Geneva, UNCTAD. [41] UNCTAD (2004a) Partnership for Development: Information and Knowledge for Development, São Paulo. UNCTAD [42] UNCTAD (2004b) E-Commerce and Development Report 2004. [43] UNDP (2006) Global partnership for development. Annual Report. New York, UNDP. [44] WorldBank (1999) Knowledge for Development. World Development Report. Oxford University Press, World Bank. [45] WorldBank (2001) Attacking Poverty. World Development Report. Oxford University Press, World Bank. [46] WorldBank (2002a) Information and Communication Technologies: A World Bank Group Strategy. Washington, World Bank. [47] WorldBank (2002b) Information Communication Technologies. A World Bank Group Strategy. Washington DC, World Bank. [48] Yin, R. K. (1993) Case Study Research, Design and Methods. California, Sage Publications.


Political Incentives and Policy Outcomes: Who Benefits from Technology-Enabled Service Centers?
Jennifer Bussell
delivery of government services such as identity cards, birth and death certificates, and licenses; the supply of welfare and redistributive goods; and general government-citizen communication (see, inter alia, [4]-[6]). Developing country governments responded to this opportunity and began to develop new systems for “eGovernance” and digital technology-based service delivery (see, inter alia, [7]-[12]). Yet, after more than a decade, the question remains whether governments have utilized new technologies to enable better service delivery to citizens. In India, the main subject of this study, state governments in nearly all of the twenty major states had implemented policies to provide technology-enabled services to citizens by 2006.1 These “technology-enabled service center” policies deliver public and private services2 to individuals through the use of information technologies, in particular computers and the Internet, at dedicated local centers. These centers provide a “one-stop shop” environment for multiple government departments, thus streamlining the process by which citizens access an array of services.3 Yet the outcomes of efforts in India, and thus the benefits new technologies provide to citizens, differ tremendously across the states, in terms of both the extent and character of services provided. In the low-income states of Chhattisgarh and Orissa, the Chhattisgarh government provides nearly forty services, while Orissa offers fewer than ten. The types of services also vary, with states such as Andhra Pradesh providing a wide range of high-demand services, while others, including Uttarakhand, provide only a few low-demand services. I argue that the observed variation in technology adoption, and the related benefits derived by citizens, result from the strategic calculations of political elites. Drawing on extensive sub-national analysis of technology policies in the Indian
1 The constitutional allocation of responsibilities in the Indian federalist system makes sub-national state governments responsible for the provision of a majority of government services to citizens, rather than the national government. As a result, states are institutionally the most appropriate level for implementing service centers. 2 Service centers often provide a mix of government services and services from the private sector, such as Internet access, digital photography, or telemedicine services. 3 While there may be related policies regarding service delivery via mobile centers or mobile phones, the policies studied here all involve service delivery at an immovable center through the use of information and communication technologies (ICTs).

Abstract—This study investigates the causes of variation in government policies to use information and communication technologies to improve service delivery to citizens. I ask why state governments in India vary in the number and type of services they offer to citizens through technology-enabled citizen service centers. I argue that politicians estimate the expected electoral benefits from providing improved services to citizens and weigh these benefits against the costs of increased government transparency and associated reductions in corrupt income. Politicians then design service center policies to maximize their chances of retaining power. Because levels of corruption and the characteristics of electoral competition vary across the Indian states, we see related variations in technology policies. These variations in policy, and in particular the services made available to citizens, have important effects on who benefits from citizen service centers. I use evidence from sixteen Indian states to test these arguments, and show that the character of the ruling government and the level of state corruption are robust predictors of variation in state-level technology policies. Index Terms—Information and communication technology, development, India, corruption, politics.



hy do some governments invest in new technologies

while others do not? Why do certain governments implement full-scale reforms of public service delivery, while others do so in a superficial manner? The emergence of new information and communication technologies in the 1990s raised these questions in a stark way. Low-cost, digital technologies offered prospects for increasing the effectiveness of government, particularly through improved service delivery to citizens. International observers expected developing country governments, often criticized for failing to deliver services in an effective and transparent manner [1], to be the prime beneficiaries of new digital technologies [2], [3]. Developing country citizens were expected to benefit from improved
Manuscript received September 17, 2008, accepted January 20, 2009. This work was supported in part by the National Science Foundation under Grant No. 0326582. Jennifer Bussell is a Ph.D. Candidate in the Department of Political Science at the University of California, Berkeley, Berkeley, CA 94720-1950 (email: jbussell@berkeley.edu).


states, I posit that politicians attempt to utilize technology initiatives to maximize their chances of retaining power, but that these policies have both potential benefits and costs. Politicians evaluate the tradeoffs between expected effects on their political interests, especially their potential for reelection, of implementing new policies. When the balance of politicians’ calculations differs across states, we observe variation in the policies these states adopt, and in the associated benefits for citizens. This paper looks specifically at the ways in which electoral politics affect the services offered in Indian state service center initiatives. I argue that electoral incentives, as shaped by the combined effects of preexisting institutions of corruption and party competition, affect the number and type of services made available to citizens. I then test this argument using data on services in the sixteen Indian states4 that implemented technology-enabled citizen service center policies during the period from 1999 to 2006.5 The analysis is based on a new dataset of state service center policies that I developed during fieldwork in sixteen states, in addition to supplementary data collection. I utilize both quantitative and qualitative analytic techniques to evaluate the factors associated with variations in technology-enabled service provision across the states. After considering the details of my argument, I address trends in the number and type of services6 made available. II. ELECTORAL INCENTIVES AND TECHNOLOGY POLICY I argue that politicians will implement technology-enabled service centers, and specific services within those centers, when the expected electoral benefits from doing so outweigh the electoral costs. The benefits to politicians are not likely to differ dramatically across states. Politicians and parties can amplify the electoral benefits of technology policies by targeting services to their most important constituencies, yet because every incumbent should be able to benefit from such a strategy, the overall size of the electoral benefit of services reform should be similar for political incumbents across states. The costs to politicians from service centers come primarily in the form of foregone corrupt income. In the preexisting process of service delivery, politicians use the machinery of the state to extract rents [14]. Yet in the Indian context they do so not simply for personal pecuniary gain but rather to enhance their hold on political power. In a cycle of
I consider only those states in which the state government implemented a service center policy, thereby excluding the Northeast states and Jammu & Kashmir, where the central government initiated a service center initiative, and the four remaining states in which no service center policy was implemented (Bihar, Goa, Jharkhand, and Madhya Pradesh). 5 In some states more than one service center initiative was implemented during the period under consideration, for example one in urban areas and a second in rural areas. In these cases I consider the overall services offered across all projects, omitting any overlapping services. 6 Evaluation of services requires an analysis of the services that are stated to be available versus the services that are actually available in the centers. While this is difficult to assess without visiting thousands of individual centers, a reasonable measure of available services can be made from a combination of site visits, interviews with project representatives, and reviews of associated websites.

funds noted first in the literature by Wade [14], citizens pay bribes to bureaucrats, who then channel some portion of these funds to politicians, either in hopes of a job transfer or simply to satisfy a political boss’ demands (see also [15]). Politicians then use a portion of these funds to finance future elections. As a result, any threat to this flow of funds is a threat to politicians’ ability to secure reelection.7 I contend that the size of these costs depends primarily on variation in two key characteristics of the Indian states: the level of corruption and the cohesion of the ruling government. The level of corruption affects incumbent politicians’ dependence on bribes as a source of campaign finance—the greater the share of bribes in overall campaign finance and thus the more dependent politicians are on bribes, the less supportive they will be of policies to increase transparency in service delivery. There is substantial variation in the level of corruption across the Indian states, with fewer than 20% of citizens in states such as Kerala and Himachal Pradesh encountering demands for bribes when interacting with government officials, while more than 60% of citizens Bihar or Karnataka have direct experience with bribing in multiple government departments [16]. The cohesion of the ruling government, by which I mean whether a single party or a coalition of parties rules in the state, can also affect costs. Coalition versus single-party rule is relevant because of the role that supporting minority parties play in the stability of coalition governments. When a large party requires the support of smaller parties to form a government, the lead party often allocates ministerial posts to its coalition partners in order to reinforce their support. In most cases, the incentive for holding a ministerial post is not control over policy, but access to the associated departmental rents. This is because ministers can demand a portion of the bribes collected by bureaucrats in their department during service delivery. Such rents provide a concentrated economic and political benefit to ministers, who can use these funds to finance re-election campaigns. When politicians consider the potential costs of new technology policies, ministers who control departments delivering a high volume of services to citizens—for example, Transportation—may then expect major costs from more efficient service provision. From the perspective of a coalition member, a threat to this income is also a threat to the expected benefits of participating in the coalition. Moreover, minority partners in a coalition government typically reap only a limited, often disproportionately small, electoral benefit, as a party, from service center projects; credit instead tends to accrue to the larger majority party. Yet because minority partners are crucial to the survival of coalition governments, and because decision-making power is decentralized across ministers from all coalition parties, key supporting party ministers often have power to resist new technologies for delivering services. As a result, supporting party ministers have both potential
7 My argument also extends discussions of India as a “patronage democracy” (Chandra, 2007; 2004) by focusing on how patronage and other funds come from corrupt, rent-generating activities by bureaucrats and politicians.


incentives to resist technology policies and important decision-making power within the coalition that they can use to influence policy outcomes. In single-party ruled states, the concentrated costs for individual ministers are often outweighed by the overall electoral benefits of technology policy to the party. Singleparty governments also place decision-making authority with a small number of individuals, in particular, in the government’s Chief Minister. As leader of the party, the Chief Minister “internalizes” a large portion of the electoral benefit to the party of implementing service center projects. In single-party governments, the electoral benefits of implementing service center projects are concentrated, particularly for the Chief Minister, while the costs are in part diffused among Ministers and other individual Members of the Legislative Assembly (MLAs) who have limited capacity to resist the directives of the Chief Minister, due largely to party discipline. In contrast to single-party governments, then, in coalition governments the political costs of service center implementation are concentrated in key decision-makers with the capacity to resist—or alter the character of—project implementation, while the electoral benefits are, from the perspective of these key decision-makers, diffused outwards. This logic suggests that the type of ruling government— namely, whether single-party or coalition—should then affect the character of new technology policies. There is qualitative evidence for the importance of both corruption and ruling government cohesion in the choice of services to include in service center initiatives. In interviews with state level bureaucrats, representatives of states with low or average levels of corruption were more optimistic that the introduction of technology would be politically feasible. A bureaucrat from Orissa, a state with just below average corruption, when asked about resistance to service centers within government, noted that “There has been resistance, but we are slowly getting rid of it…People understand now that the story has gone past where they can stop it” [17]. Whereas an official from Haryana, a state with relatively high corruption, stated that, “many departments are not moving forward…with the implementation of front-to-back eGovernment services” [18]. He felt that this was largely due to the threat of increased transparency. For example, in “the Transport Department people in the department do not want to shift to a new system because they have established ways by which they are able to skim money off the top and they don’t want to lose these sources of income” [18]. Analysts have also noted the importance of coalition governments for service center outcomes in individual state cases. Kiran [19] comments that a service center project in the state of Kerala “was opposed by the participating departments on account of the fear that they would lose their existing authority and power. This was particularly evident in Kerala, which is ruled by a coalition government, with different political parties in charge of different departments.” Thus the policy-making incentives for party leaders come from the potential electoral benefits of implementing “good governance” technology policies and the contrasting threats

from decreased corrupt income. I now consider the effects of these political incentives on the character of technologyenabled services. III. CORRUPTION, COALITIONS, AND TECHNOLOGY-ENABLED SERVICE DELIVERY A key question for an analysis of service center policies is which citizens actually benefit from introduction of these technologies. Because state governments are responsible for providing services to their entire population, if they offer technology-enabled services that benefit only a portion of the population, then they are failing to offer the same benefits to other citizens. Fig. 1 shows the variation across Indian states in the number of services made available to citizens.

Fig. 1 – Available Services in the Indian States (2006) My argument offers specific predictions for the ways in which politicians’ calculations over the likely costs and benefits of technology policies may affect who benefits from these policies and to what degree. In cases where the expected costs from technology adoption are high, such as in states with high levels of corruption and in states ruled by a coalition government, governments are likely to implement policies in ways that serve fewer citizens overall. A. The Quantity of Services I evaluate the relationship between level of corruption and policy outcomes using a novel measure of state-level bureaucratic corruption, which is uniquely suited to testing my theoretical claims. This measure draws on a Transparency International survey of corruption in India [16]. The survey asked citizens about both their experience with corruption in acquiring services from government and their perception of corruption in government. Transparency International provides an indexed corruption score by state, based on eleven departments, including the police, municipal services, electricity, and the judiciary. I use this indexed score to develop my measure of bureaucratic corruption across the Indian states. The survey is particularly appropriate for the purposes of this analysis because it focuses explicitly on bureaucratic corruption in low-level service delivery, the area


targeted by service center policies, rather than the high-level corruption more often engaged in by top officials. In states with high levels of corruption, the large anticipated costs from more transparent service delivery should discourage service provision overall, leading to fewer services than in lower corruption states. The cohesion of the ruling government should also affect service selection, as coalition governments are likely to resist the inclusion of supporting-party controlled services, leading to fewer services overall. In this section, I discuss descriptive statistics, differenceof-means tests, and OLS models I use to evaluate these hypotheses. It is worth bearing in mind that the sample size is small: with only sixteen state cases, statistical power may be low. If I am able to uncover statistically significant relationships, it is because these relationships are particularly strong.8 Corruption First, the level of corruption in a state is a strong predictor of the number of government services offered in computerenabled service centers. The major findings of this analysis are presented in Table I. Among the states with below average bureaucratic corruption levels, the mean number of services offered is 20.1. This noticeably contrasts with the mean number of services offered in states with above average levels of corruption, which is 10.6. This difference is statistically significant at the .05 level. As noted above, any significant finding with only sixteen cases implies a particularly strong relationship between the variables under consideration. A univariate regression model offers similar findings. The relationship between quantity of services and corruption, when corruption is measured on a scale from zero to one, is not statistically significant, but is in the predicted direction. When corruption is measured dichotomously, the relationship is significant at the .01 level (t-statistic of -3.22) (See Model 1 in Table III below). Table I Indian States, Corruption, and Quantity of Services Corruption Below Average Above Average Absolute score: 240-478 Absolute score: 479-695 Scaled score: 0-.523 Scaled score: .524-1 Andhra Pradesh (41) Delhi (6) Chhattisgarh (37) Haryana (10) Gujarat (19) Karnataka (22) Himachal Pradesh (24) Rajasthan (11) Kerala (11) Tamil Nadu (10) Maharashtra (16) Uttarakhand (2) Punjab (16) Uttar Pradesh (13) Orissa (7) West Bengal (10) Mean services: 20.11 Mean services: 10.57 Difference in means: 9.54 (t-statistic = 2.07)
Statistical power, of course, is a function not just of the sample size but also the effect size.

Ruling Party Cohesion The next relationship to consider is between the type of government and the quantity of services. Are single party governments associated with higher numbers of services? Here, the evidence is not as clear-cut. On average, states ruled by single parties do offer more services than coalition states; with a mean of 18.0 services while coalition states have a mean of 11.4 services. The difference between these two means however is statistically significant only at the .1 level. Given the small number of cases and the fact that the services do trend in the direction predicted by my argument, however, it is reasonable to believe that there is a meaningful relationship in the data between the quantity of services and the type of government. In the case studies below I discuss additional evidence for a relationship between government cohesion and services outcomes. Corruption and Government Cohesion Perhaps the more relevant test of my argument is an analysis of the relationship between corruption and government cohesion. The effect of corruption on the number of services is most evident in low corruption, single-party states. These states provide, on average, a larger number of services than any other states, and this difference is statistically significant. The effects of ruling government cohesion and high levels of corruption are somewhat more difficult to parse, as the average number of service provided in high-corruption, singleparty states is not statistically different from that provided in coalition-ruled states. Coalition-governed states provide fewer services on average than single-party states, regardless of the level of corruption in the state. Thus, based on this data, it is only possible to say that either coalition states will provide fewer services regardless of the level of corruption, or that high corruption states will provide fewer services than low corruption states regardless of ruling government cohesion. In order to provide greater clarity on the role of ruling government cohesion in service selection, I evaluate the specific services offered by coalition governments in detail below.


Table II Mean Number of Services by Corruption and Ruling Government Cohesion Ruling Difference Government in Means, Cohesion Single Single Coalition Party Coalition Party (t-statistic) Corruption Low High Difference in Means, Low - High Corruption (t-statistic) 27.4 10.2 17.2 (3.04) 11.3 11.5 -0.2 (-0.06) 16.1 (2.87) -1.3 (-0.43)

Table III Multivariate Analysis of Candidate Explanatory Variables Variable Model 1 Model 2 Model 3 Model 4 Constant 24.7 25.8 24.5 24.9 Above Average Corruption Ruling Party Cohesion Months Since Initiation State Domestic Product per Cap (7.19) -14.0** (-3.22) (6.85) -13.7** (-3.12) -3.5 (-0.80) (2.29) -13.3* (-2.39) -3.4 (-0.75) .02 (0.14) (2.30) -12.4 (-2.17) -4.2 (-0.88) .1 (0.38) -9.0

Alternative Explanations The length of time elapsed since a service center initiative was implemented might reasonably be expected to influence the number of available services. As governments in states that implemented centers early will have had more time to introduce additional services. The level of economic development might also be associated with provision of services, as states with higher incomes might have more funds available to invest in government reforms. In order to test these alternative explanations, I used multivariate regression models to evaluate the relationship of level of corruption, ruling government cohesion, time elapsed since implementation, and state economic development with the quantity of services available in the states. As shown in Table III, the level of corruption shows a clear and robust relationship with the quantity of services, even when taking into consideration these alternative explanations. However the length of time since centers were initiated and the level of economic development in a state show no relationship with the quantity of services. Ruling government cohesion also does not show a statistically significant relationship with quantity of services, but as noted above this may in part be due to the small number of coalition states in the dataset.

(-0.86) N 16 16 16 16 r2adj .38 .37 .32 .30 Entries are unstandardized regression coefficients with t-ratios in parentheses. *p < .05 **p <.01 B. Selection of Services The benefits of computerized service centers to citizens depend not just on the quantity of services, but also on the character of the specific services made available. In this section I consider first whether high demand and high corruption services are made available in services centers across all states. I then consider the specific types of services made available in coalition-government led states. The government of India, and in particular state governments in India, provides hundreds of services to citizens on a regular basis. Table IV lists the fifteen most commonly provided services in computerized centers across the states. Out of the 73 government services offered by at least one service center initiative, only these services are offered by more than 25% of the projects (shown in Table IV).9 I also include one service, ration cards, that is offered in 24% of initiatives, but is interesting to include in the analysis because it is provided by a different department, Food and Civil Supplies, than any of the other services. Are these services in high demand by citizens? Based on Transparency International India’s survey of corruption in public service delivery [16], only a small number of those services most needed by citizens are provided in computerized service centers. Of the top twenty-five services required by citizens, thirteen are offered in at least one state. However only six of these are among the services offered in at least 25% of states (highlighted in bold in Table IV). Thus, in many cases there is a disconnect between the services that are provided by states and those that are in high demand by citizens. This disconnect is most obvious in states with above average levels of corruption. High corruption states provide,
Birth and death certificates, which are offered by 62% of initiatives, is the only service provided in more than 50% of states.


on average, 2.9 of these thirteen services, while below average corruption states provide 5.2, a difference of means that is statistically significant at the .05 level. When the number of high demand services is regressed on the level of corruption, the relationship is in the predicted direction and significant at the .05 level (t-statistic of -2.33). Table IV Most Common Services offered in Indian Computer-Enabled Service Centers Department Service Municipal Corp/Rural Dev Grievances Birth & Death certificates Caste/tribe certificates Property tax payment Ration cards Land records Income Certificates Residency certificates Electricity bill payment Telephone bill payment

Food & Civil Supplies Revenue

Electricity Telephone Company Water Transport

Table V Availability of High Corruption Services in Indian ComputerEnabled Service Centers Reason for Est. % of < Average > Average Bribe Annual States Corruption Corruption Bribes Offering States States ($ ‘000) Police First 458,000 13% 2 0 Info. Report School Fees 340,000 0% 0 0 (exemption) Rural 313,000 0% 0 0 Financial Loan Register 307,000 50% 5 3 Property School 288,000 0% 0 0 Certificate Electricity 166,000 13% 2 0 Connection Estimated annual bribe value based on Transparency International India data [16] and author’s calculations. In the case of coalition-led states, I also expect to see effects of electoral considerations on the specific services that are chosen for inclusion in centers. Services controlled by supporting members of a coalition government should be less likely to be included than those services controlled by the majority party. Figure 2 provides an overview of service provision in coalition-led states. As shown in this graph, politicians from the smaller, supporting members of a coalition are more likely, on average, to acquire ministerial posts that involve control over services commonly offered in service centers. This is in line with expectations that supporting coalition members will demand ministerial appointments in departments with high levels of government-citizen interaction. However, when supporting coalition members control these services, it is less likely that they will be included in service centers, as seen from the right side of the graph. In the case studies below I consider the specific ways in which allocation of ministerial posts is linked to these lower levels of service provision.

Water bill payment Driving licenses Vehicle registration Home/Police Arms license Bold indicates that this service is one of the top 25 most demanded services by citizens, according to [16]. An alternative measure of citizen demand for service computerization is the estimated level of corruption in a particular service. Because bureaucrats are more likely to demand a bribe for some services than others, service reforms that have the potential to reduce corruption should provide greater benefits to citizens when introduced for services with a high likelihood of corruption. However reform of these services will also provide the highest costs to bureaucrats and politicians in terms of lost income, and so computerization in high-corruption potential services should be least likely in high corruption states. When we consider high corruption services, separate from their overall demand by citizens relative to all services, states with higher levels of corruption are less likely to provide these services through computerized service centers. As seen in Table V, below average corruption states are much more likely to provide services associated with bribe payments than states with above average corruption levels. Out of the six most corrupt services, only one, land records, is available in any of the above average corruption states.


Figure 2 – Average Service Provision in Coalition States – Lead vs. Supporting Parties Kerala I begin the case discussion with Kerala. This state is viewed by many as an innovator in both development in general [20] and ICT-enabled development in particular [21], [22]. Despite below average income per capita, Kerala has the highest literacy rate in the country and the highest overall human development index [23], [24]. Kerala also applied its development strategies to the use of technology and was one of the first states to use IT to reform service delivery (Kurian, 2003). Yet the services offered in Kerala’s two main service center initiatives, FRIENDS and Akshaya, pale in comparison to many of their counterparts in other states. Through FRIENDS centers, only ten government services are offered, with just three offered in Akshaya centers. Could coalition politics be at least partly to blame for the low quantity of services in Kerala? As noted above, analysts have pointed to the importance of coalition dynamics in shaping Kerala’s service center initiatives [19]. Once the first service center initiative was approved, the FRIENDS initiative, the ruling Left Democratic Front (LDF) continued to wield their influence over the shape of the centers. This coalition was led by the Communist Party of India (Marxist) (CPI(M)), with its coalition partner the Communist Party of India (CPI) holding the next largest number of seats in the state assembly during the period from 1996 to 2001. The coalition in total held 76 seats, five more than the necessary majority of 71. Ministerial posts were allocated across coalition members, with the largest number of ministries predictably allocated to the CPI(M). Six of the fourteen most common services were made available through the FRIENDS centers: electricity bill, telephone bill, water bill, property tax payment, driving licenses, and vehicle registration. Of these services, two of the relevant departments were overseen by supporting coalition members, the Water Department and Transport, led by the Revolutionary Socialist Party (RSP) and an Independent member of the legislature, respectively. Because the RSP held only five assembly seats and the independent, by default, held

only one, neither of the two ministers from these parties had the power to bring down the coalition government on its own: even defection by all five RSP ministers would leave the coalition with 71 seats, sufficient to maintain its majority. The remaining services offered through FRIENDS centers were delivered by departments overseen by CPI(M) ministers. Among the common services not offered through these centers, two key departments, Revenue and Food & Civil Supplies, were overseen by the CPI, the only party with enough assembly seats to threaten the stability of the ruling coalition. The other relevant departments, Local Administration and Rural Development, were both led by representatives of the CPI(M). The lack of services offered by departments under the CPI(M)’s most important coalition partner aligns clearly with predicted outcomes. The Akshaya project, a second computer-enabled center initiative in Kerala, offers a similar story of coalition politics, with an interesting twist. The Akshaya initiative was launched under the United Democratic Front, a Congress Party-led coalition that came to power in 2001. In this coalition the Congress held the largest number of seats, at 62, but needed the support of one or the other of its two main coalition partners, the Muslim League of Kerala or the Kerala Congress (M), to reach the magic number of 71 for a majority. Akshaya was launched in 2003 through a pilot project in the state's Malappurum district. The stated goal of the initiative was to increase access to technology for the citizens of Kerala, and a key part of the initiative was computer literacy, or “e-literacy,” for the head of every household, a goal that mimicked Kerala’s well-publicized success in boosting traditional literacy rates (www.akshaya.net).10 But these centers provided an additional opportunity for the delivery of government services, as noted by those affiliated with the project [25] Overall, however, the Akshaya project has implemented next to no services through its network of what in 2008 is more than 1,000 rural centers. Only three common government services are on offer: electricity bill payment, telephone bill payment, and grievance redressal. The Electricity Department is overseen by a Congress minister, and the Department of Rural Development, which would oversee grievances in the rural areas where Akshaya centers are located, is headed by a representative of the Kerala Congress (M). While it is surprising that the KC(M) would have one of their services offered through the Akshaya centers, it is perhaps more telling to note the number of services overseen by the Kerala Congress (M) that are not offered through these centers. During the UDF government, the KC(M) was in control of both the Revenue Department and Rural Development, which together account for six of the remaining commonly offered services, none of which have been provided through Akshaya. Other coalition parties controlled the departments overseeing nearly all other common services,
At nearly 91%, as measured in the 2001 Census, Kerala has the highest literacy rate of any Indian state or Union Territory.


with the Kerala Congress (Jacob) and the Kerala Congress (B) overseeing Water and Transport, respectively. Congress ministers oversaw only two common services that were not offered via Akshaya, arms licenses and ration cards. For a portion of these services, the lack of provision through Akshaya is even more surprising, as these services had already been implemented at the district level in FRIENDS centers. Thus even though the basic infrastructure was in place to extend water bill payments, property tax payments, driving licenses, and vehicle registrations through Akshaya outlets, this was not done under the Congress administration. This means that citizens must either go to the district FRIENDS office or avail of these services through traditional department-based offices, which are still be prone to any corruption in the system. The twist in the Akshaya case is the way in which the policy itself was used as a tool to target the core voters of a key coalition member party. The Malappurum district where the pilot project was initiated has a largely Muslim population, and the Muslim League is the dominant party in the region [26]. When the UDF coalition regained power in 2001, a prominent member of the Muslim League, Shri. P. K. Kunhalikutty, became Minister for the Information Technology Department. According to analyst reports, the Minister subsequently consulted with representatives of Malappurum and determined that information technology could be a “real enabler of the local economy” [26]. In order to facilitate IT use, a plan was developed by the local government to provide for computer literacy in the district. This plan was subsequently modified in coordination with the Kerala State IT Mission, under the oversight of the IT Minister, and realized as a “telecenter” initiative for the district [26]. If successful, this pilot would then be extended to other districts in the state. So while the stated intention was to develop an initiative that could eventually benefit the entire state, the initial benefits would clearly be centered on the core constituency of the standing IT Minister, representing a key ruling government coalition member. Akshaya, then, provides an important example of the combined effects of coalition rule. First, a second computer center initiative in the state was launched for the apparent purpose of rewarding key Muslim League constituents, a clear boon for the holder of the IT Ministry post. Second, the lack of services offered through Akshaya reflects the prevalent use of ministerial posts by the UDF government to reward coalition members. Even those services that should be relatively easy to implement because of their inclusion in the earlier FRIENDS initiative have, for the most part, not been made available to Akshaya patrons. Karnataka The next state, Karnataka, has a much higher level of corruption than Kerala, which would, on its own, imply a stronger emphasis among coalition members on the economic side-benefits of ministerial posts. But Karnataka is also a “hard” case, because it is one of the most technically advanced

states in the country, with the IT hub of Bangalore as its capital. On the face of things, Karnataka has also been at the forefront of using ICTs in service delivery. The Bhoomi land records initiative received more media attention than most initiatives in the country, and the former Secretary for eGovernance, Rajiv Chawla, is widely recognized as a father of the Indian movement to incorporate information technologies into Indian government processes. Yet, as we will see, even the presence of early initiatives and an important evangelist do not guarantee de-politicized service delivery in the state. A major service center initiative, Bangalore One, was initiated under the coalition government of the Congress Party and Janata Dal (Secular) (JD(S)) in 2005. This particular government was formed despite the fact that the BJP held the largest number of seats in the assembly, because the BJP was short of a majority and could not agree to a coalition with other parties to acquire a majority [27]. So while the Congress has the greater number of seats in the coalition, it is highly dependent on the support of its JD(S) partner.11 Bangalore One was launched in partnership with the municipal government in Bangalore and was intended to provide a similar one-stop government services environment as offered by Andhra Pradesh’s eSeva initiative [28]. In the case of Bangalore One, however, despite multiple years of experience with technology-enabled service delivery, many of the most commonly offered services are not available. Of the three services that fall under the domain of the Revenue department—residency certificates, income certificates, and land records—none are offered in the Bangalore centers. The exclusion of land records is especially surprising, given that the Bhoomi land records initiative was viewed as such a ‘success’ in the state. Why are these common and highly valued services not offered in Karnataka? The most plausible answer again lies in coalition politics. The minister in charge of Revenue was a politician from the supporting JD(S), who also held the post of Deputy Chief Minister, and so was clearly an important player in maintaining the coalition. Ration cards are also unavailable in the centers, and it was a minister from the JD(S) who oversaw the responsible Food & Civil Supplies department. Of those common services that were offered, the JD(S) was responsible for only one, electricity bill payment, while the Congress was responsible for the remaining services, via Municipal Administration, the department that was also involved in the implementation of the initiative itself. So even in a state with a strong emphasis on technology and history of government IT initiatives, the dynamics of corruption and coalition politics seem closely linked to the selection of specific services made available in the state’s one-stop centers. Rajasthan
11 The coalition eventually collapsed in early 2006, when a rebel JD(S) leader, H.D. Kumaraswamy, pulled out of the government with a group of other JD(S) MLAs in order to form an alternative coalition with the BJP (Rediff, 2006).


In Rajasthan, the LokMitra (urban) and JanMitra (rural) centers were opened in early 2002 under the leadership of the Indian National Congress party. During this period the Congress party held a majority of the seats in the state assembly and so did not require the support of any other parties in a coalition. The initiation of service centers was largely an enterprise of the Chief Minister Ashok Gehlot, who sent representatives of the state to Andhra Pradesh in 2001 to evaluate the eSeva initiative there and determine how something similar could be used to improve service offerings to citizens in Rajasthan [29]. According to one of the bureaucrats who participated in that mission, a decision was made to improve on the Andhra model by offering services in both urban and rural areas, through two separate initiatives, at least at the beginning.12 The choice of services in each set of centers was then determined based more on urban versus rural needs, with the LokMitra centers focused on bill payments and JanMitra centers emphasizing grievances and non-government services such as agricultural prices. However, in rural areas only four government services were offered, while in urban areas just seven services were made available. The availability of specific services, particularly in urban areas, seems directly linked to the interests of the Chief Minister. When bureaucrats in charge of implementing LokMitra services attempted to convince various departments to provide their services in the centers, they faced significant resistance. The state telephone company did not want to allow outsourcing of its bill payments through the computerized centers, and the bureaucratic officers found it necessary to request the intervention of the Chief Minister. In this case the Chief Minister did intervene and the Telephone company was forced to provide services for bill payment through the LokMitra centers. On other occasions, however, such as in the case of income tax payments, the Chief Minister was unwilling to take similar initiative and income tax payments were not included in the services offered by the centers [29]. The mixed response of the Chief Minister in the case of Rajasthan may be closely tied to the high level of corruption in public service delivery in the state. As one bureaucrat involved in the initiatives noted, politicians “want to provide the maximum services to their vote bank,…[but] they also want money for the next election” [29]. Thus from the perspective of the Chief Minister, by providing some services, especially those that do not typically involve high levels of corruption, such as bill payment, there is an opportunity to reap some political benefits from the centers. At the same time, those services that may more often involve the payment of bribes, such as income tax, can be restricted from inclusion in the centers in order to maintain the rents from those services. In this way we can see how high levels of corruption may affect service delivery in single party states. Chhattisgarh The final case to consider is Chhattisgarh, a single-party-led state that has below average levels of corruption in public
JanMitra and LokMitra were merged into a single initiative, eMitra, in 2004. Both Jan Mitra and Lok Mitra are Hindi variations on the term “people’s friend”.

service delivery. The state of Chhattisgarh was formed in the year 2000 when it was carved out of Madhya Pradesh. Despite below average levels of economic development and a persistent and violent separatist movement in part of the state, the Chhattisgarh government has implemented a substantial eGovernment program through its Choice service centers. These centers, which to date are largely located in urban centers, provide one of the largest sets of services of any state in India. Why is it that Chhattisgarh has implemented such a robust service center initiative? Here again we see the important role played by Chief Ministers in the character of public service reforms. Choice service centers were initially implemented under the first Chief Minister of the state, Ajit Jogi of the Indian National Congress. However the growth of the project from a pilot stage has occurred under BJP Chief Minister Raman Singh. According to government officials associated with the Choice initiative, Chief Minister Singh encouraged an inclusive decision-making process in which the bureaucrats in charge of major citizen-facing departments would participate in an “empowered committee” that was tasked with making all key decisions about the project moving forward. The goal of this model was to ensure consensus on decisions so that no participants would later attempt to block implementations of the centers or any specific services [30]. In addition, the Chief Minister took over leadership a “governing council” made up of government ministers, which is responsible for overseeing the implementation of the project. This meant that the implementation team could go directly to the Chief Minister in case of any problems with implementation. In no other state, other than perhaps Andhra Pradesh, has the Chief Minister taken on this type of direct oversight role in the development and implementation of service centers. What factors might have influenced the Chief Minister to take such a strong position on eGovernment in the state? Corruption does exist in Chhattisgarh, but government representatives argued that the expectation of the Chief Minister was that improvements in service delivery would provide a greater potential electoral boost to the ruling government that any threat from reduced access to rents. “The Chief Minister sees the benefits as greater than the costs. He has been the brainchild behind all of these frameworks. Indian politicians…have very sharp political minds. The Chief Minister in this term is focused on good governance, on accountability, transparency, and responsiveness of the government” [31]. Given the difficulties that the government otherwise faces in areas of development and social stability, it seems that the Chief Minister has adopted service reforms through the use of technology as an important platform for delivering valued goods to citizens. This is possible, at least in part, because he does not likely feel the same threat to campaign resources from increased government transparency that exists in higher corruption states. IV. CONCLUSION The evidence presented here provides strong support for an argument about the role of corruption and coalition dynamics


in affecting which citizens benefit from technology-enabled service delivery. Citizens in more corrupt states are less likely to benefit from new technologies. This is because even if service centers are implemented in their state, these centers are likely to provide fewer and lower demand services than those centers in other states. Citizens may also be at a disadvantage depending on the characteristics of their ruling government. In coalition-led states, state governments will chose services that ensure the stability of the government before considering what services might benefit voters. When MLAs from supporting parties serve as ministers for departments, the services of those departments are less likely to be implemented, particularly if that party holds enough seats to disrupt the majority of the coalition. Who benefits from technology-enabled service centers is thus highly dependent on the political characteristics of a state and in particular the extent of corruption and cohesion in the ruling government. While there may be great potential to improve service delivery through one-stop service centers it is clear that, at least to date, the actual benefits to citizens are politically driven and thus, in the case of the Indian states, highly varied. REFERENCES
[1] [2] [3] [4] [5] [6] [7] World Bank, World Development Report: Making Services Work for the Poor. Washington, D.C.: The World Bank and Oxford University Press, 2004. World Bank, ICTs and MDGs: A World Bank Group Perspective. Washington, DC: The World Bank Group, 2003. Accenture, Markle Foundation, UNDP, Creating a Development Dynamic: Final Report of the Digital Opportunity Initiative. 2001. R. M. Davison, C. Wagner, and L. C. K. Ma, “From government to egovernment: a transition model,” Information Technology and People, Vol. 18(3), pp. 280-299, 2005. United Nations, World Public Sector Report: E-Government at the Crossroads. New York: United Nations Publications, 2003. D. West, Global E-Government 2005. 2005. Available: http://www.insidepolitics.org/egovt05int.pdf J. Bussell, “Electoral Competition and Digital Development in India and South Africa,” Proceedings of the IEEE/ACM International Conference on Information and Communication Technologies for Development 2007 (ICTD2007), 2007. K. De Tolly, B. Maumbe, and H. Alexander. “Rethinking E-Government for Development: Issues, Lessons and Future Prospects for the Cape Gateway Portal in South Africa,” presented at the IST-Africa Conference, Pretoria, South Africa, May, 2006. S. Madon, “Evaluating the Developmental Impact of E-Governance Initiatives: An Exploratory Framework,” The Electronic Journal on Information Systems in Developing Countries, 20(5), pp. 1-13, 2004. J. Satyanarayana, EGovernment…the science of the possible. New Delhi: Prentice-Hall, 2004. R. Heeks, “Causes of eGovernment Success and Failure: Factor Model,” 2003. Available: http://www.egov4dev.org/causefactor.htm S. Bhatnagar, “Lessons from eGovernment in Developing Countries” Regional Development Dialogue, No. 24, 2002. Government of West Bengal, IT Department official, January 18, 2008. R. Wade, “The Market for Public Office: Why the Indian State is not Better at Development,” World Development, Vol. 13(4), pp. 467-497, 1985. F. De Zwart, The Bureaucratic Merry-go-round: Manipulating the Transfer of Indian Civil Servants. Amsterdam: Amsterdam University Press, 1994. Transparency International India, India Corruption Study 2005. New Delhi: Transparency International India, 2005. Government of Orissa, IT department official, January 29, 2008.

[18] Government of Haryana, IT Department official, February 4, 2008. [19] G.R. Kiran, “Front-End First: Citizen Payment at FRIENDS Centres in Kerala,” UNPAN eGovernment for Development Success/Failure Case Studies, Number 17, 2002. Available: http://www.egov4dev.org/friends.htm [20] P. Heller, The Labor of Development. Ithaca: Cornell University Press, 1999. [21] S. Madon, “Governance lessons from the experience of telecentres in Kerala,” European Journal of Information Systems, Vol. 14, pp. 401416, 2005. [22] S. Madon, and G.R. Kiran. Information technology for citizengovernment interface: a study of FRIENDS project in Kerala. World Bank Global Knowledge Sharing Program (GKSP), 2002. [23] Reserve Bank of India; GOI, 2001 [24] Reserve Bank of India; GOI, 2005. [25] Kerala Government, IT Department official, June 22, 2004. [26] G.R. Kiran, “Akshaya, Malappuram, Kerala” in Information and Communication Technologies for Development: A Comparative Analysis of Impacts and Costs from India,” Bangalore: International Institute of Information Technology, 2005. [27] Rediff.com, “JD-S, Congress Team up in Karnataka,” May 16, 2004. Available: http://ia.rediff.com/election/2004/may/16karna.htm [28] Karnataka Government, eGovernance Department official, February 22, 2006. [29] Government of Rajasthan, former IT Department official, May 7, 2007. [30] Government of Chhattisgarh, IT agency official, January 23, 2009. [31] Government of Chhattisgarh, IT Department official, January 23, 2009.


[9] [10] [11] [12] [13] [14] [15] [16] [17]


Results from a Study of Impact of Egovernment Projects in India
Subhash C. Bhatnagar and Nupur Singh

Abstract—The paper presents the results from an assessment study of eight e-government projects from India. The assessment framework measured the total value delivered by a project to various stakeholders on three dimensions: (a) cost to clients for accessing services; (b) perception of quality of service and governance; and (c) agency cost and revenue. Data was collected from randomly selected users encapsulating their experience of using the computerized and manual systems. Impact was estimated as the difference between the rating of the computerized and the manual systems. Citizens indicated an overwhelming preference for computerized service delivery. The number of trips required to be made to the concerned offices reduced significantly and waiting time came down by nearly fifty percent. Overall impact showed wide variations across projects. Of the seven agencies where corruption was reported in the manual system, five services saw significant reduction through computerization but the impact was marginal in three projects. Government of India has adopted the framework used in this study to assess the impact of 40 projects implemented at the national, state and local levels. Paper discusses the implications of the results for further development of e-government projects in India and identifies the limitations of the study. Index Terms—E-government, governance, impact assessment, India

are making ICT investments in the public sector. On the other hand, evidence of failed projects has drawn attention to the level of risk involved in implementation. A failure rate of more than 50 percent is widely cited in this context [1]. A study commissioned by DFID [2] researched firm-level impact of ICT in developing countries, 2 but similar studies have not been undertaken for the public sector. A report by the United Nations (2003) laments the fact that documented research on the social or economic impact of e-government development is virtually non-existent [3]. Multilateral organizations such as the World Bank fund nearly seven billion dollars of ICT investment as part of their lending programs and grant assistance to various client countries. A World Bank report noted that the largest yet the least monitored investments are IT components of projects in different sectors, highlighting the relevance of systematic assessment of the impact of these applications [4]. This paper presents an early effort at evolving a systematic framework and methodology for assessing the impact of egovernment projects based on a review of past efforts at assessment of e-government projects. Sections II and III discuss the review and present the key features of the framework. The paper goes on to report the findings from a study that used the framework to ascertain the impact of a selection of eight e-government projects from India. II. EFFORTS TO ASSESS PUBLIC SECTOR IT PROJECTS



N recent years a large number of countries have launched “e-government” programs, and several development agencies and governments have identified e-government implementation as a key policy priority. 1 Driven by the success of a few projects in improving delivery of services to citizens and businesses, an increasing number of governments
Manuscript received February 5, 2009. S. C. Bhatnagar is with the Indian Institute of Management, Ahmedabad, India (phone: 91-79-6632-4802; fax: 91-79-6632-6896; e-mail: subhash@iimahd.ernet.in). N. Singh is with the Centre for E-Governance, Indian Institute of Management, Ahmedabad, India (e-mail: nupurs@iimahd.ernet.in). 1 Examples of e-government initiatives launched by international agencies can be found on the following links: World Bank (http://www.worldbank.org/egov); ITU (http://itu.int/itudoc/itu-t/workshop/egov); and DFID (http://www.publictechnology.net/modules.php? op=modload&name=News&file=article&sid=1533)

While significant amount of academic and policy research has focused on evaluation of public sector ICT projects, a systematic framework for evaluation has not emerged. One group of studies provides macro-level estimations of egovernment activity using appraisal indices focusing on supply-side, quantifiable measures such as web presence of government, network coverage, institutional and regulatory support and human capital provision. 3 Such factor-based assessments of e-readiness do not have immediately obvious and tractable policy implications. They tend to focus almost
2 The study sponsored by DFID at the London Business Schools collected data from firms in India and Brazil to establish a relationship between ICT investments, profitability, growth and productivity at the firm level. The degree of collateral organizational change was studied as a mediating variable. 3 Examples of such frameworks are (1) UNPAN (2004) E-Government Readiness Report www.unpan.org and (2) Brown University (2004). Global E-Government Report, available on www.insidepolitics.org/egovt05int.pdf


exclusively on measurement of physical access to certain types of ICT without incorporating issues such as affordability, appropriateness, ICT capacity and training, and the regulatory and macroeconomic environment [5]. The second group of studies, largely anecdotal, done in a piecemeal fashion, provide project-level evaluations with little prospect for synthesis from past approaches. Evaluative studies have been done to serve a variety of purposes. Some studies looked at implementation success in terms of whether the systems were functioning as they were designed to, or the degree to which the intended outcomes were achieved. Some studies looked at long term sustainability and scope for replication, 4 while some measured the benefits that were delivered to agencies. 5 A few focused on benefits to the clients. 6 There was hardly any comprehensive study that assessed the impact on all the stakeholders and covered both short-term and long-term direct and indirect impacts. A few studies had carried out a cost-benefit analysis. Often, evaluation studies had been done by agencies that were likely to have an interest in showing a positive outcome. A variety of approaches have been used for evaluation. These included surveys, expert opinion, ethnographic studies and internal assessments produced by lending agencies. The utility of such evaluations has been limited because: ▪ Different studies of the same project showed very different outcomes, thus indicating a lack of credibility of results. 7 Part of the reason for different outcomes was the use of very small samples and a lack of rigor in sampling and collecting data from clients of the systems. The results could therefore not be easily generalized over the entire population of clients. The studies evaluated the functioning of the computerized system but were not able to assess the difference made by ICT use, as the need for counterfactuals (evaluation of systems as they worked before computerization) was ignored. Often, the impact of ICT use was not separated from other interventions that were made simultaneously with the computerization effort. Finally, since different studies did not use a standard methodology, it was difficult to compare the outcome of one project with that of others.

creation in e-government projects have been documented in reports by governments and external consultants [6] – [8]. An example is the study conducted jointly by the Danish Ministry of Finance and Accenture which identifies two primary dimensions of value: (i) value to investors in terms of tangible financial benefits, cost savings, cost avoidance, and increased revenue; and (ii) value to users in terms of improved services, reduced cost and/or time savings to citizens, and reduced administrative burden to businesses. Investment in government employees and technological infrastructure are identified as key enablers for creating value through egovernment. The focus has been on using business case methodologies from the private sector to demonstrate the economic value of e-government projects. The calculation of time and money spent in finding and using public information are the most direct and measurable benefit of e-government applications to users, as shown in recent OECD and EU studies [9],[10]. These reports identify improved revenue collection, lower costs due to efficient processing of transactions, and a reduction of administrative burdens due to simplification or elimination of procedures as some direct impacts that can be measured. Financial savings to users in terms of time and money spent in finding and using public information are the most direct and measurable benefits of egovernment applications for clients. A significant amount of work has also been done on studying users’ perceptions of quality in terms of attributes such as accessibility, attitude of staff, cost of service, provision of information, procedural fairness and convenience. Calculation of net economic benefit of an IT project has proven to be challenging as outcomes are multi-dimensional and composed of both quantitative and qualitative indicators. Further attention is required to be given to the linkages amongst issues of quality of service, governance, wider impacts on society, and ICT investment. Newer frameworks are evolving around a notion that an exclusive focus on financial costs and benefits for the government ignores many important non-economic benefits [9]. One such benefit relates to improvements in various aspects of governance activity. Another non-economic benefit relates to the addressing of wider policy priorities, which, in the context of developing countries could constitute the priorities as articulated in the UNDP Millennium Development Goals. III. MEASUREMENT FRAMEWORK AND METHODOLOGY USED IN THE STUDY Two approaches - MAREVA developed by the ADAE (Agence pour le Development de l’Administration Electronique - Electronic Administration Development Agency) in France with the help of Bearing Point and the WiBe Economic Efficiency Assessment methodology being used by the German federal administration [11] were useful in developing the framework proposed in this chapter. These methodologies developed by two Governments in EU countries focus broadly on the same dimensions. They offer two levels of impact assessment: first, in terms of how the project provides a business case justification of expenditure

Work done in EU countries has focused on understanding the ‘processes’ that deliver a return on investment and generate value [6]. Common best practice factors for value
4 Government of India, Ministry of IT has commissioned quick assessment of 29 projects. 5 Korea’s eProcurement agency has evaluated the impact on different government agencies using the system. 6 Global Knowledge Sharing Program got 4 Indian projects evaluated where clients were surveyed. 7 For example the Bhoomi project of issuing copies of land title has been evaluated by Public Affairs council reporting a significant positive outcomes including reduction in bribes. Recent studies by a team from MIT and IIIT Bangalore found that corruption had not declined and major benefits were derived by land sharks.


and second, in terms of how the project meets the goals of the agency concerned and, in turn, how this helps in meeting wider government strategies. Guiding these assessment frameworks are strategic outcomes pursued by respective governments representing broad policy priorities that drive the direction of government. These considerations have been incorporated in the assessment framework proposed for egovernment projects in this section. 8 Borrowing from the above approaches and the work done in EU countries, the framework used for the study reported in the paper focused on the idea of measuring the total value delivered by a project to different stakeholders and takes a balanced approach between case study and quantitative analysis. It recognizes that some part of the value for each stakeholder can be monetized and other part needs to be assessed qualitatively. Most of the assessment models discussed in the literature have not been used in the context of a developing country and many have not yet been applied in practice even in developed countries. Nor do they account for a variety of delivery models used in developing countries such as common service centers and franchised outlets that can retail e-services offered by the government. Practical issues of paucity of data have not been taken into account, particularly in case of a developing country where baseline surveys are not done and monitoring and evaluation systems are weak. Recognizing that adequate data for quantitative assessment is not available for most of the projects, collection of such data using a standard measurement framework for sufficiently large number of projects is the first task that needs to be undertaken. Since impact assessment is the key objective, establishing counterfactuals is an important element of the proposed measurement framework. A number of empirical studies suggest that ICT has had an impact in improving the performance of private sector organizations particularly in developed countries. However, regarding ICT investments by the public sector in developing countries, many researchers have noted that past evaluation studies have not used a common framework or methodology and that rates of success/failure have been declared based on purposive samples [12]. Since ICT is introduced mostly in the context of governance reform to improve efficiency, effectiveness, and transparency of governments, a crucial first stage is to ascertain to what extent these intended outcomes from e-government applications have been achieved. A common measurement framework evolved on the basis
The indicative items are based on a review of the following documents: i) Performance Reference Model of the US Federal enterprise Architecture framework used by the office of Management and Budgets in US Federal Government; ii) European Commission, eGovernment Economics Project (eGEP), Measurement Framework Interim Version Deliverable (D.2.2), 2005; iii) TP Rama Rao, V Venkata Rao; SC Bhatnagar; J Satyanarayana, ‘eGovernance Assessment Frameworks’, 2004, http://egov.mit.gov.in; iv) Edwin Lau, ‘Electronic Government and the drive for growth and equity’, OECD, 2005; and v) Lanvin Bruno, ‘METER E-strategies Monitoring and Evaluation Toolkit’, in Robert Schware (ed), E-Development: From Excitement to Effectiveness, Prepared for the World Summit on the Information Society Tunis, 2005, World Bank.

of existing frameworks has been used in this study. An eservice delivery project impacts three groups of stakeholders: (i) clients receiving the service; (ii) agency (including implementation partners) that delivers the service; and (iii) the larger society consisting of citizens, businesses, government as a whole and civil society. The impact can be assessed in terms of a variety of outcomes experienced by each type of stakeholder. Table I lists key dimensions of outcomes for each type of stakeholder.
TABLE I KEY OUTCOME DIMENSIONS Stakeholders Client Key Dimension of Impact ▪ Economic (Direct and indirect) ▪ Governance (Corruption, accountability, transparency, participation) ▪ Quality of service (Decency, fairness, convenience, etc.) ▪ Economic (Direct and indirect) ▪ Governance (Corruption, accountability, transparency, participation) ▪ Performance on key non-economic objectives ▪ Process improvements ▪ Long term impact on Millennium Development Goals ▪ Image of the government

Agency (Including partners in implementation)

Society Government as a whole

The primary objective of the study was to measure the impact of computerization on clients (users) of selected service delivery projects and to test the applicability of the framework across a variety of projects. For the purpose of this study a sample of eight mature projects shown in Table II was selected. These projects covered services to rural as well as urban citizens (G2C), services to businesses (G2B) and services for internal government users (G2G). Most of the services are offered by state-level agencies except in eSeva where services from the federal government are also offered. In eProcurement services are accessed through a portal. In the remaining projects service delivery is through assisted computerized counters set up by the agencies at various locations. For each project, the measurement framework was converted into a set of data collection instruments including: i) a profile of the project identifying services, clients and other stakeholders; ii) agency level data on activity levels, investments and operating costs; iii) a client survey questionnaire covering direct cost of access, quality of service and governance, and a few measures of overall satisfaction; and iv) an employee survey for perceived impact on work, efficiency and effectiveness. The survey assessed both the manual system and the computerized system that replaced it on all the above dimensions. An analysis of the differences between the old and the new system provided a measure of impact. Random samples of about 30 users were chosen fro