Sustaining peer‐based learning through reputation modeling
Version - June 2nd , 2010 Thieme Hennis Supervisors: Wim Veen & Stephan Lukosch
Index 1 Introduction 1.1 Developments in open education 1.2 Gift culture, reciprocity, and reputation 1.3 Semantics of reputation 1.4 Problem statement 1.5 Outline of the proposal 2 Literature review 2.1 Learning 2.1.1 Learning theories and pedagogies 2.1.2 Learning scenarios 2.1.3 Conclusions 2.2 Sustainability 2.2.1 Sustainability of Open Educational Resources 2.2.2 Sustainability of online communities 2.2.3 Conclusions on sustainability 2.3 Reputation and trust 2.3.1 Why reputation? 2.3.2 Trust and reputation 2.3.3 Context, transitivity, and cross‐community reputation 2.3.4 Examples of trust and reputation systems 2.3.5 Conclusions on trust and reputation 2.4 Semantic web 2.4.1 What is the Semantic Web? 2.4.2 Semantic Web Technologies 2.4.3 Relevance of the Semantic Web 3 Scientific approach: philosophy, strategy, and planning 3.1 Research questions 3.1.1 Operational research questions 3.2 Societal and scientific relevance 3.3 Research philosophy and methodology 3.3.1 Ontology and epistemology 3.3.2 Methodology 3.3.3 Summary of research philosophy 3.4 Methodology: Design Science 3.4.1 Guidelines 3.4.2 Design Science activities 3.5 Design cycles and instruments 3.5.1 First cycle: developing the semantic model 3.5.2 Second cycle: developing the reputation system 3.5.3 Research instruments 3.6 Scope of the research 3.7 Transferability, acceptance and dissemination of research 2 4 4 6 7 8 9 10 11 11 16 17 18 18 21 22 24 24 25 26 28 34 38 38 39 40 41 41 42 43 44 46 46 47 48 49 50 51 51 53 55 57 57
3.8 Planning List of references
In 2008, George Siemens and Stephen Downes organized an online course on Connectivism and Connective Knowledge. Over 2200 persons worldwide actively participated in an online, peer-based learning network. Feedback is an essential feature in learning. Because the organizers were unable to assess and give individual feedback to each of the students, they motivated students to give peer-feedback, using technologies of all kind: virtual worlds, blogging, commenting, RSS feed-readers, Moodle CMS, and much more. Interestingly, this course was given for free, as part of a research project by the two organizers. Since 2001, open education has been a linked with all kinds of initiatives that have the objective of providing better access to learning by providing learning resources for free on the Internet. The course by Siemens and Downes, together with several other similar initiatives the last couple of years, adds another element to the open education spectrum: guidance and peer-support. Connectivist and networked learning approaches focus on the ability of the network as a whole to learn, and to learn from the network. CCK08 was an interesting initiative, but only partially shows how learning is evolving with the Internet. Even though it gave access to learning resources, and provided guidance, it offered official accreditation to only a very limited group: only 25 (less than 2 percent of the students) could be evaluated. Traditional methods for evaluation and assessment seem inadequate for an online context.
1.1 Developments in open education
The figure below shows the developments in Open Education.
Learning communities Free online courses
• CCK08 • Guidance by experts • Peer‐assessment • Limited accreditation • Tools to support social interaction and knowledge exchange • P2PU.org • Peer‐based learning • Tools to support peer‐ assessment and open accreditation?
Open Educational Resources
• Example: MIT OCW • Static and dynamic content • Tools to publish content
Figure 1 - Developments in Open Education
Two important challenges are identified for the open education community:
sustainability to the continuous ability to create, maintain, and share high-quality learning resources[6,7,8,9,10,11,12], and assessment, accreditation, and recognition of learning achievements in a scalable and sustainable way[4,5,13,14,15,16].
The proposed research intends to provide solutions to both of these challenges. As the learning landscape is gradually transforming into more networked, online, and interactive learning landscapes[1,3,5,10,17,18,19,20,21,22], we find ourselves facing significant challenges on different levels. The roles of learning institutions is changing rapidly[15,21,22], new skills are needed that emphasize on the learners’ capability to find relevant information scattered in (online) networks [4,18,22]. Technology (most significantly, the Internet) offers many new opportunities for learning and learning institutions, but important questions need to be answered: “If we are going to imagine new learning institutions that are not based on the contiguity of time and place—virtual institutions—we have to ask, what are those institutions and what work do they perform? What does a virtual learning institution look like, who supports it, what does it do? We know that informal learning happens, constantly and in many new ways, because of the collaborative opportunities offered by social networking sites, wikis, blogs, and many other interactive digital sources. But beneath these sites are networks and, sometimes, organizations dedicated to their efficiency and sustainability. What is the institutional basis for their persistence? If a virtual site spans many individuals and institutions, who or what supports (in practical terms) the virtual site and by what mechanisms?” We are currently moving toward a new future of open education and learning. Networked, informal learning in online communities of practice is argued to become an important mode of learning [23,24,25,26,27,28,29]. Participation in these online communities also have positive influence on professional life: “…through the process of engaging in a VLC, individuals may change their ‘horizons of action’ leading to new learning and career trajectories. In particular, the study demonstrates how membership of a VLC supported and enabled some individuals to transform their learning careers and to make significant life changes. Other members developed their learning careers in an incremental manner that led to increased innovation and professional expertise.” If we turn our attention to the concerns mentioned by Davidson and Goldberg , we identify a need for new models of organization of learning. Recent developments on learning theories and pedagogies, including social constructivism[27,31,32,33], Connectivism[4,16,21], problem-based learning, communities of practice[23,24,28,29,35,36], active and networked learning[37,38,39,40,41], self-regulated learning[42,43,44,45,46,47,48,49] describe learning as a intrinsically social activity that can be supported by online communities.
1.2 Gift culture, reciprocity, and reputation
Addressing the opportunities of learning in online communities, we must focus on the question how these communities come to existence and are sustained. Inspiration may come from the Open Source domain. In these self-organizing, online communities, responsibility to locate, assemble, and contextualize learning resources is with the members themselves. Although the tragedy of the commons  would suggest that such voluntary collaborations are not sustainable over time, the emergence of the Internet, and specifically the Free/Libre Open Source Software (FLOSS) movements, have shown peer-to-peer communications technology’s ability to put people in symbiotic, “you answer my question, I’ll answer yours” relationships. The gift culture described by ethnographers of the FLOSS movements such as Raymond (1999), Himanen (2001), Benkler (2005), and von Hippel (2003) is one explanation of this phenomenon[51,52,53,54]. Another reason for ‘free’ forms of knowledge exchange is one of knowledge management: expertise exists across the community, so no individual community member is overly burdened with the primary responsibility for answering questions and providing feedback. As problems arise related to the expertise of an individual, that individual may or may not choose to provide help. If the community is of sufficient size, the distribution of expertise and effort provides timely problem solving support without unduly burdening any individual. The inventor of the term ‘gift economy’, Marcell Mauss (1926), describes the gifts as a particular form of bonding, of creating mutual dependencies: The form usually taken is that of the gift generously offered; but the accompanying behavior is formal pretence and social deception, while the transaction itself is based on obligation and economic self-interest. Reciprocity, or the expectation to get something in return, is a central motivation in online communities, where ‘gifts’ take the form of sharing expert knowledge, or the creation of digital artifacts[25,56,57,58]. This important driver for knowledge exchange and participation in online communities brings us to the question what kind of ‘return on investment’ a community member expects when he or she shares knowledge. Learning, in its most fundamental form, is aimed at survival. Darwin once famously said that not the most intelligent, or strongest of species would survive, but the one most adaptable to change. Even though adaptation is a form of intelligence, one could argue, the importance of learning is made clear. It is a means to be able to cope with changes in the environment. For most human beings, learning is aimed at getting in tune with society. Higher education is aimed at preparing learners for professional work[2,59,60]. A rapidly changing society therefore requires learners that can adapt quickly, and learn relevant things on demand, whilst being able to de-learn irrelevant things[17,43]. Reciprocity, in the scenario of lifelong learning in online communities, will take the form of getting relevant information on demand, and developing a reputation that is useful in a professional context [4,14,16,25,36,44,51,58,61,62,63].
Again, the Open Source community can be used as an example of how online reputation can be used as an artifact to convey trust in a person, both for employers and between developers [41,64,65,66,67,68,70]. Tuomi describes it as follows: “For example, the social importance of formal educational certificates is now declining, as the capabilities, interests, and reputations of people can be directly evaluated using information and communication technologies. Instead of asking whether a job candidate has a formal educational status, a potential employer can now review the candidate’s actual track record, blog postings, and possibly e-portfolios. Already today people compile extensive life histories using web logs and homepages. In some expertise areas, for example in computer programming, employment opportunities often depend on a track record that can be reconstructed by search engines and personal blogs. The digital identities of persons now consist of their own representations of achievements and experiences, as well as reputations that accumulate through the comments of others.” Reputation modeling in online communities could provide an online alternative to institutional assessment methods and as an incentive for individuals to engage in all roles needed to sustain peer-based learning environments. The question is then: “how can, and should reputation be modeled in order to provide trustworthy representations of knowledge and skills (as institutional accreditation aims to do), and provide incentive to share knowledge and (partially) sustain peer-based online learning?” There are two basic criteria for a reputation to be useful. First of all, the profiles that are created in an online reputation system must be accurate. Secondly, the community as well as the external world must recognize (or trust) the reputation. These requirements lead us to the Semantic Web.
1.3 Semantics of reputation
The Semantic Web is a large-scale effort to define terms, concepts, and relationships on the Internet in machine-readable way, unambiguously[71,72,73]. Systems are being designed that can use semantically annotated data in various ways, including semantic search, matchmaking, automation, analysis, and more. The requirement of accuracy and external validity of the proposed reputation system points in the direction of the Semantic Web as a relevant domain to position the research (for development and publishing). By definition, a reputation system that adheres to open semantic standards can more easily be used in other contexts than a closed system. Acceptance and applicability are requirements for an open reputation system, and following Semantic Web standards seem an obvious path to follow. Some relevant existing semantic schemas describing trust and reputation have been developed[63,74,75,76,77,78,79,80]. Despite the developments in this direction, none of the schemas describes the interactions specific to peer-based learning in online communities. This research will find out which elements in which schemas can be reused in order to make a useful conceptualization of reputation that supports peer-based learning. Reusing schemas and ontologies improve the likeliness of acceptance and prevent redundancy[81,82].
1.4 Problem statement
With cheap or freely available technologies to create, disseminate, and evaluate information, as well as social networks and specialized communities, the Internet is considered highly suitable for active and networked learning. Whereas there are models in place to ensure supply of learning resources and even guided courses free of charge, there is no model for ‘open’ assessment and accreditation of achievements in online communities. An online reputation system that assesses learning (including problem-solving and teaching) in online communities can be used to improve trust in expertise and capabilities of individuals based on their actions and achievements online. Such a system needs to be developed in order to improve access to the whole spectrum of education (Art.26 – UN Declaration of Human Rights); from learning resources, to guidance, to assessment, to accreditation. This research will develop a reputation system to support the assessment of learning achievements in online communities. The reputation system will take into account newer learning approaches and intends to model the ‘value’ of objects (including persons) to support learning. The solution may be inspired by various online initiatives that have a similar focus: increase trust in objects by means of measuring social interactions with those objects. These systems, one of which is Google, have not only proven to be very useful, but also commercially sustainable. A system that is able to recognize and interpret peer-assessment in learning communities can be used to accredit, in some way, the achievements of an individual learner, and prove his/her value. The direction we take is aimed at modeling learning interactions and knowledge exchange in online communities. The assumption of the research is that interactions and knowledge exchange activities can be logged, and that these interactions contain information about someone’s expertise and value in online communities. This information can be used in a professional context, if a member chooses to share his/her reputation information with relevant third parties, such as potential employers. Recognition by 3rd parties is an important incentive for individuals to participate and interact in a way that increases their reputation.
1.5 Outline of the proposal
The following chapter is the literature review, which explains the above reasoning in more detail. • The first section will focus on the relevance of peer-based learning in online communities. It concludes with a number of challenges, and an argument for the development of a standardized reputation system aimed at peer-based learning. Reputation systems, focus of the second part of the literature review, have been developed to increase trust in and quality of objects, improve interaction and support transactions between people, and as an incentive to be active in a specified way. They can therefore provide important lessons for the development of a scalable and sustainable system that needs to do the following: assess the quality of learning resources, show the value (and expertise) of individual people, and sustain ‘good’ or sustainable behavior in peer-based online communities. Using exemplary reputation systems, it concludes by stating that interoperability and openness of the reputation system will increase the likeliness of adoption. Developments on semantic web technologies have created a useful framework to start from, and that much of the work in this domain can be reused for the purpose of this research. The final chapter is aimed at explaining the Semantic Web, and defines possible trajectories that can be taken in order to create a useful, and accepted model for reputation on the Internet.
The third chapter will describe the research philosophy, and defines the research questions. The last chapter consists of a detailed planning of the research.
2 Literature review
In the literature review, each main section starts with an argument why the section is relevant. The section ends with a few lines summarizing the conclusions and relevancy. The literature review connects learning theories and newer opportunities enabled by the Web, with motivation and sustainability. This is followed by an extensive analysis of reputation, being an important incentive for knowledge sharing, and thereby a potential source for sustainability of online peer-based learning environments. The relevance of the Semantic Web in creating a valid and scalable reputation system is explained.
The Internet and the numerous online communities of practice and professional networks provide opportunities for informal, self-regulated and networked learning. Above all, the open character offers relatively cheap access for individual learners worldwide to connect with people and find relevant content. The Internet is an environment in which skills can be developed that are needed in a technology driven, and rapidly changing society. The skills that learners develop in regular education systems are different from those developed in peerbased communities[26,29,84]. The first part of this chapter will focus on learning theories and the relevance of the Internet for learning. Because the research is focused on improving sustainability of peer-based learning in online communities, a basic understanding of learning and pedagogies is needed. In addition, if we intend to sustain peer-based learning through semantic modeling of learning activities, it is important to know which activities constitute peer-based learning. The second part of this chapter will focus on the issue of sustainability in open education environments and online communities. Whereas we have systems in place, developed over centuries, that sustain learning in traditional environments, the systems in place that sustain learning in peer-based online communities are still very much in their infancy. The coming decades, we will see the emergence of institutions and systems that sustain high-quality learning in peer-based online communities. In order to create sustainable online environments for learners to engage in peer-based learning, we must recognize and address both the opportunities and challenges facing us. Quality management, assessment and recognition, and motivation to collaborate or to share information are just a few of the challenges[10,12,18,44,85,86,87,88]. The final part of this chapter describes the role of social mechanisms, including reputation, with respect to these challenges. It concludes with an argument that many of the challenges for sustaining peer-based learning in online communities can be influenced with a reputation system. 2.1.1 Learning theories and pedagogies Below, a brief overview of the development of learning theories is given, based on Bransford et al. . Drawing on the empiricist tradition, behaviorists conceptualized learning as a process of forming connections between stimuli and responses. Motivation to learn was assumed to be driven primarily by internal drives, such as hunger, and the availability of external forces, such as rewards and punishments (e.g., [89,90]). A limitation of early behaviorism stemmed from its focus on observable stimulus conditions and the behaviors associated with those conditions. This orientation made it difficult to study such phenomena as understanding, reasoning, and thinking—phenomena that are of paramount importance for education. Over time, radical behaviorism gave way to a more moderate form of behaviorism that preserved the scientific rigor of using behavior as data, but also allowed hypotheses about internal "mental" states when these became necessary to explain various phenomena. In the late 1950s, the complexity of understanding humans and their environments became increasingly
apparent, and a new field emerged—cognitive science. From its inception, cognitive science approached learning from a multidisciplinary perspective that included anthropology, linguistics, philosophy, developmental psychology, computer science, neuroscience, and several branches of psychology [91,92]. New experimental tools, methodologies, and ways of postulating theories made it possible for scientists to begin serious study of mental functioning: to test their theories rather than simply speculate about thinking and learning (see, e.g., [92,93,94,95], and, in recent years, to develop insights into the importance of the social and cultural contexts of learning (e.g., [96,97,98,99,100]. The social nature of learning Constructivism is the psychological theory, which argues that humans construct knowledge and meaning from their experiences [101,102,103]. Constructivist educational theory focuses on concept development and deep understanding, rather than behaviors or skills, as the goals of instruction. Personal development and deep understanding happens through the construction of meaning by the learner self, not through transmission from one person (the teacher) to another (the learner). The fundamental principle of constructivism is that learners actively construct knowledge through interactions with their environment . Therefore learners are viewed as constructing their own knowledge of the world. For effective learning, knowledge should be uniquely constructed by people through play, exploration and social discourse with others. Learning objectives presented in constructivist learning environments should be firmly embedded in context, and should, at least in some way, represent every day life situations. Learners should also accept responsibility for their own learning and be self-motivated to explore different knowledge domains. The central point of social-constructivism is an individual's making meaning of knowledge within a social context . Learning as a social practice is well established and dialogue is one of the corner stones of social constructivism. This makes online communities such potentially effective places for learning. The interactions in online communities is being maintained through a sense of community and social capital through information flow, altruism, reciprocity, collective action, identities, and solidarity to support the development of democracy[64,65,107,108]. These are central elements that need attention in an online social learning context. Social mechanisms that address internal cohesion and sense of community are important for learning and overall sustainability of a social learning environment, but so are mechanisms that impact interaction with the external environment, including reputation and recognition. Learning in communities The term ‘situated learning’ locates learning in the process of co-participation and in the field of social interaction, not in the head of individuals to get an inter-subjective understanding and meaning of something. In communities, learning means moving from the peripheral (lurking, being introduced into processes, people, etc) into the center (sharing expertise, making decisions). Peripheral participants do not accumulate knowledge and skills but are
introduced in processes, routines, networks, relevant issues, and approaches within the community. “The individual learner is not gaining a discrete body of abstract knowledge which (s)he will then transport and reapply in later contexts. (…) There is no necessary implication that a learner acquires mental representations that remain fixed thereafter, not that the ‘lesson’ taught consists itself in a set of abstract representations” . Edgar Dale developed the so-called “learning cone” or learning pyramid . It shows the retention level in different learning activities, ranging from reading (low retention), to hearing, to seeing to saying and doing (high retention). Although the research has been criticized , it remains a powerful metaphor for communicating the effectiveness of active types of learning. Learning as knowledge creation is seen as the epistemological foundation of CSCL, Computer Supported Collaborative Learning. Paavola, Lipponen and Hakkarainen explain the “knowledge-creation” metaphor of learning as follows; “Learning is seen as analogous to processes of inquiry, especially to innovative processes of inquiry where something new is created and the initial knowledge is either substantially enriched or significantly transformed during the process”. Hence, learning goes beyond the information given. Since traditional models of distance learning have not inspired researchers and teachers to develop innovative pedagogical practices, current research and development work in the field has turned towards creating multi-faceted pedagogical practices, utilizing ICT, that can support learners in their efforts to engage in deeper-level learning and interaction. Allert writes that in modern knowledge societies, there is a need for scenarios that focus on collaborative processes of creating innovative knowledge. This type of learning comprises of open, ill-structured problem solving processes, focuses on communication and collaboration. Stahl emphasizes that meaning is collaboratively produced in a cultural context, embodied in a physical or semantic artefact, and is situationally interpreted within a community or social system. He refers to learning as shared meaning making, which is not understood as a psychological process which takes place in individuals' minds but as an "essentially social activity that is conducted jointly - collaboratively -- by a community, rather than by individuals who happen to be co-located". Meaning is not transferred from one thinker to another, but is constructed. Processes of knowledge construction and shared meaning making happen increasingly in virtual environments, such as games, online communities and forums. CSCL aims at supporting this type of learning through the design of powerful learning and communication environments integrating collaborative learning and the use of ICT . Learning with understanding Historically, education has focused more on memory than understanding. An emphasis on understanding leads to one of the primary characteristics of current theories of learning: its focus on the processes of knowing [101,103]. Humans are viewed as goal-directed agents who actively seek information. They enter a learning process with a range of prior knowledge, 13
skills, beliefs, and concepts that significantly influence what they notice about the environment and how they organize and interpret it[97,98]. This, clearly, can have both positive and negative consequences for the learning process and their abilities to remember, reason, solve problems, and acquire new knowledge. Effective learning environments, effective support systems for learning, and effective teachers therefore take into account the background of a learner. Control over learning New developments in the science of learning also emphasize the importance of helping people take control of their own learning. Since understanding is viewed as important, people must learn to recognize when they understand and when they need more information. Effective learning environment therefore focus on sense-making, self-assessment, and reflection on what worked and what needs improving[4,48,84,112]. Networked learning Learning is becoming a lifelong, self-directed and collaborative effort, in which one engages with people and finds resources online. Learning institutions should focus on supporting this process, and guide students in assessing and evaluating knowledge they encounter online. Leaders at learning institutions need to adopt a more inductive, collective pedagogy that takes advantage of the collaborative and participative spirit of our era and the potential of the internet to connect people, link information sources, and support creativity. Rather than individual learning based on competition and hierarchy, is a more networked model of learning preferred, because it allows learning from peers, and stimulates cooperation, partnering, and mediation. Veen, Lukosch and de Vries describe a pedagogical approach for networked learning, presented below.
Figure 2 - Networked Learning Model (Veen, 2006)
The ingredients of the Networked Learning model can be seen in the above figure. There are four complementary areas that play an important role in knowledge development. Each of the 14
elements that are connected to these areas is relevant for this development process in which the technology is a major facilitator for processes of communication, information retrieval and information sharing. These areas are: Profiling, Connectedness, Knowledge and Business Development. • ‘Profiling’ states that individual users should take ownership of their professional development, ICT enabling them to do this through social software tools. A way for teachers of profiling is act as a tutor (individual online support, for example in forums), coach (general support on specific areas), or scaffold (provides handhelds for students to bring them further), and instructor (writes instructions and manuals). Students can also profile their presence as helper and peer-tutor, or as a critical but just evaluator of learning materials. ‘Connectedness’ stands for the connection between people and people and resources. It relates to social networks and the way interaction and human relations are relevant for people to perform in communities. These communities are fluent; you can take part for some time depending on the purpose of the community. Communities are based on peer references and are not limited to office hours. ‘Knowledge’ is the area that defines content and information in the Network Learning Model. This content is distributed and discontinuous, stored in databases. Learners have to aggregate bits and pieces (modules) into a meaningful whole. They do this collaboratively, sharing their expertise with others. ‘Business Development’ is the area that describes the major companies’ business goals, what they offer and for what purpose. These goals are the reference framework within which learning takes place, it provides the organizational context (e.g. Dept. of Physics, Univ. of Thessaloniki).
Networked learning focuses on interconnectedness between people and between people and resources [37,38,39,40]. Technology is used to integrate delivery of knowledge with interaction, communication and application. The earlier mentioned concept of Communities of Practice  is integrated in Networked Learning, because learning practices and social practices are interconnected, the learning practices emerge from participants rather than be imposed by facilitators, learners are involved in concrete practical actions together, learning is not designed, rather designed for, variation in levels of expertise can expand the group’s learning, networked learning needs to support visits to “otherness”. Connectivism Widely adopted learning theories behaviorism, cognitivism, and constructivism, and combinations of them, do not sufficiently explain the effect of technology in our lives and learning activities. George Siemens and Stephen Downes have attempted to explain learning in a digital age by combining and enhancing different learning views, and developed Connectivism[4,21,115]. An important distinction from social constructivism is the emphasis on the fact that knowledge does not need to be internalized and emphasizes that learning also happens outside a person’s mind. Siemens argues that in the Information Age the learning process concerns activities such as synthesizing and recognizing patterns, meaning making, and forming connections between specialized communities. Know-how and know-what is
supplemented with know-where as the understanding of where to find the knowledge needed. Connectivism addresses learning outside the person, knowledge stored in databases or other electronic information holders accessible through the Internet. It describes a form of knowledge and a pedagogy based on the idea that knowledge is distributed across a network of connections and that learning consists of the ability to construct and traverse those networks. This implies a pedagogy that seeks to describe 'successful' networks, as identified by their properties, such as diversity, autonomy, openness, and connectivity; and seeks to describe the practices that lead to such networks, both in the individual and in society. Connectivism extends the notion of learning as a personal, internal change  to a network change: Non-human elements act as actors in the network and the medium itself is part of wider networks. A network that is able to adapt and learn is much more valuable than a static network; therefore, if a person is connected to the dynamic, adaptive network, even though no internal process occurs, in this respect the learner is a better learner. 2.1.2 Learning scenarios As we can see, there is no one definition for learning. In the context of this proposal, learning is considered active, participative, and networked. Below, three short scenarios describe how learning can occur according to this learning paradigm. • In the first scenario, we assume learning to be a social activity that happens in communities, where learners take different roles, depending on context and expertise. In that case, if it would be a perfectly organized, large, and diverse community, each person would be able to find a more knowledgeable peer, who, in theory, would be willing to guide his or her learning, because that is a learning activity as well (explaining to others). Another scenario would be that several persons, through social interaction and online discussion, combine their expertise to create new knowledge. Through social interaction, they collaboratively come up with answers to questions, or develop new ideas. Another scenario, in the Connectivism paradigm, could describe how a learner tunes his network, consisting of peers and several tools, such that at any moment, he or she is able to know-where to find the relevant knowledge. This requires tools to manage knowledge and information streams, but also to find relevant knowledge outside your own network.
How can we design communities in order to sustain this learning behavior? How do we motivate people to engage in learning activities? How do we support the learner that the learning activity is in line with his or her background, such that the right person or the right resource can be found? All these scenarios emphasize the importance of finding the right person, the right tool, and the right content at the moment you need it. Finding consists of two parts: searching and storing. You may be a very good finder, but if something is not stored at the place where you look for it, you will never find it. On the Internet, if a document is not linked, you will not find it. Keywords can be used to specify the search, and help the search engine interpret your search. Finding is also related with storing. Organizations used to store documents in enormous filing cabinets, using taxonomies and armies of librarians to make sense of the information being 16
stored. Most databases and computerized file systems still function in this way, static and not able to cope with a lot of information. Google, the most successful search engine, does not store information and websites in this way. In chapters 2.3 & 2.4 (on reputation systems and the Semantic Web), a more elaborate explanation is given on possibilities of improving the way information is defined and stored, increasing the likeliness that it is used. 2.1.3 Conclusions Different approaches to learning give opportunities to make online communities selforganizing and effective places for learning. There are some important requirements; • • • Social interaction leads to learning, sense of community, and trust Problem solving and knowledge creation processes are central to both learning and the creation of new open educational resources Learning is a networked activity, which includes people and technologies in the network. The ability to find people just in time, when you need it, is an essential feature, which can be learned, but also supported through technology. The ability to find resources, people, answers, depends on the way information is stored in the network. In order to improve self-organization of learning in online communities, new ways of storing and retrieving information must be developed, specifically created for these communities. An online reputation is a data profile. Such a profile contains information about someone’s contributions and interactions in online communities. This information may be rich enough to predict the value of the person to solve a problem or teach someone. Because persons are central in a learning process, such information needs to be stored in a way that it can be found. Background information is very important in learning. As learners bring their personal history, bias, cultural characteristics, and expertise to a learning arena, reputation information that describes expertise can be used to improve the learning experience for individual learners. Reputation can also be used in order to stimulate people to engage in learning processes. If relevant third parties recognize reputation, then it forms an important incentive for lifelong peer-based learning.
The new science on learning sees the Internet as an opportunity to support peer-based learning in online communities. Several initiatives have been started recently, but sustainability is still a big issue[6,7,8,13,117,118]. The next section will focus on the sustainability of open learning communities.
Although the meaning of sustainability depends on the context in which it is used, some generic things can be said. The term is often used to refer to the ability of ecological systems to be usefully productive. Sustainability links present with future, because choices made now should not compromise the opportunities or possible benefits in the future. A typical example of the opposite of sustainability is illustrated by a recent decision by CITES not to ban fishing on blue-fin tuna1. Through overfishing the blue-fin tuna population is driven to extinction, ruining ecosystems, and taking away opportunities for future generations. Principles of sustainable development include (i) dealing cautiously with risk, uncertainty and irreversibility; (ii) integration of social and economic goals in policies and activities; (iii) equal opportunity and community participation; (iv) a commitment to best practice; (v) the principle of continuous improvement; and (vi) the need for good governance. Sustainability does not concern solely ecological and environmental systems, but relates to the continuity of economic, social, and institutional aspects of human society, as well as the non-human environment. Economic sustainability relates to the extent to which end-users rely on subsidies or financial inputs, institutional sustainability addresses the effect changes have on the social structures and institutions . 2.2.1 Sustainability of Open Educational Resources With regards to sustaining OER projects, there is no direct relation with nature or ecology. In this context, sustainability really focuses on economic and institutional sustainability. Wiley (2006) defines sustainability of an OER project as the ongoing ability to meet the goals of the project. This implies (i) continuously being able to produce and share Open Educational Resources; and (ii) to sustain the use and reuse of their Open Educational Resources by endusers. In other words, it means the ability to produce, share, localize, and learn from open educational resources. This does not necessarily depend on top-down funding, but peer production and reuse of OER can create sustainability. Wiley claims that sustainability can be reached through the reduction of friction and decentralization, capturing intrinsic motivation of individuals to contribute without financial recompense. He literally states that decentralization means the active involvement of students. Decentralization happens through the peer production of open resources and sharing them in P2P (peer-to-peer) networks; “It seems to me that sustainability and scalability are problematic only when people rely on others to do things for them. Scalability and sustainability happen more readily when people do things for themselves. Centralizing open educational services is less scalable/ sustainable. Decentralizing them is more scalable/ sustainable. Wikipedia has two employees and well over a million articles in multiple languages. We need to learn this lesson if open education is really going to reach out and bless the lives of people.” This resonates with other research on sustainability of OER projects. Koohang & Harman (2007) describe OER communities of practice as a means for decentralization and better scalability . The sustainability of Connexions, a project that almost entirely depends on the
voluntary efforts of individuals worldwide for the production of OER, has been investigated as a case study. The case study focuses and describes issues regarding motivation and value propositions for end-users of the site. Benkler describes criteria for (decentralized) peer production of OER and its positive relation with sustainability. Schmidt & Surman state that the focus should be much broader than just on content: the sustainability of OER depends on the thriving of a whole OER-ecosystem. For instance, the process of building and nurturing peer production communities should be taken more seriously. More focus should be on creating stronger communities of practice in open education. Stephen Downes follows the same argument, saying that the centralized model uses more resources, and is likely to cost more, but offers more control over quality and content. This, in fact, does not necessarily means lower quality. We have seen that in Open Source Software, and also for online encyclopedias. Reasons for sharing educational resources An important question to be answered is of course; why? What are the reasons for institutions and individuals for spending resources for giving away something that used to be competitive advantageous? Reasons for involvement of institutions include altruism (knowledge should be available for all), publicity & marketing, improving peer-network, and search for new business models using open educational resources. The most common reason for participation of individuals is the access to the best available educational resources and to have more flexible materials. The four most important barriers for involvement in production of open content, according to the OECD/CERI study on Open Educational Resources, are a lack of (i) time, (ii) reward system, (iii) skills, and (iv) a viable business model. Hylén (2007) describes three challenges of the OER movement: copyright issues, quality assurance, and sustainability. Since 2005, various highly useful open licenses have been developed, most famously the different Creative Commons license. Regarding quality assurance, three approaches are mentioned, which concern the institution as final responsible organ (closed and centralized), quality assurance by means of peer review (open and centralized), and the final (open and fully decentralized) approach, as discussed above. Quality of open educational resources In an earlier paragraph, we defined learning as shared meaning making, how do we define the quality of this process and its outcomes? Quality management is defined as the structures, activities, and processes that are designed for planning, assuring, improving and/or evaluating the quality of an institution, a product, or a service . The last couple of decades, significant attention has been given to quality management approaches in e-learning. Markus Wirth provides an extensive account of various initiatives worldwide and concludes that there is an immense amount of offerings worldwide for quality management in e-learning. He concludes that most of these initiatives and frameworks address traditional certification and accreditation, and rarely do include newer quality approaches that focus on more innovative learning. He points out that a useful categorization and systematization of these approaches should help in comparing the enormous variety of approaches. He also stresses the importance of context, because different quality approaches come from, and are based on different contexts. Leaving out this context makes the whole effort redundant. Furthermore, he
mentions various contributions in international journals and books that show considerable reservations towards the feasibility of quality management in e-learning. Traditional quality standards in higher education mislead institutions to imitate classical face to face trainings instead of fostering innovative approaches in learning and the use of (multi)media. Based on Wirth, this proposal suggests quality management in online peer-based learning communities to be defined as “the structures, activities, and processes that are designed for the creation, planning, assurance, improvement and/or evaluation of peer produced learning materials and interactions in online communities that lead to shared understanding”. More specifically, this means • • • • • the creation of learning resources, the use of them, and the discussions about these resources; recommendations learning resources, creation courses or sets of resources, and interactions that lead to structured learning, such as guidance or mentoring; retrieving, storing, and analyzing feedback and understanding user learner interactions; assessment of learning activities and contributions, and possibly some form of accreditation; and offering incentives, support and motivation to members of online communities engage in peer-based learning activities, like guiding peers or adapting resources.
Open Source Assessment and Accreditation Online resources, including Open Educational Resources, are needed, and sharing information on the Internet is almost the easiest thing in the world. What needs to be developed, are the systems and business models that truly sustain all the processes of online, peer-based learning, not just the creation and sharing of educational resources (which is a very effective way of learning). The systems and business model must be able to motivate individuals and organizations to participate in these processes. The introduction describes several initiatives that have extended the notion of open education to offer not only the resources for free, but the structures to learn as well, including support (through peer evaluation and by use of networked technologies). In one case, over 2000 students participated in a networked, online learning experience using a multitude of media and technologies. One important thing that struck me was that official recognition could only be offered to only 1 percent of the students, and for a fee. Recognition of online activities through a traditional, formal accreditation scheme is not scalable in the online context. This is in line with Schmidt et al. (2009), who identifies key challenges of learning in open education communities: recognition, accreditation, and assessment . Recognition is the acknowledgment of achievements and conveys approval by a person, group, or organization. It can be done implicitly (for example, citing a paper) or explicitly (recommending a person on LinkedIn). Accreditation is formal certification by a community, institution, or organization who thereby explicitly state that the receiver meets the standards of the accreditor. Accreditation applies to individuals as well as to institutions and to programs. Assessment is the process of determining the characteristics of something or someone. For 20
learners, assessment means determining their individual knowledge and/or skills, which is a necessary basis for recognition or accreditation[124,125]. Making an analogy with open source communities, Schmidt et al. describe four criteria for assessment and accreditation in open online learning communities: • Trust: If a community or group as a whole is not trusted, the quality and reputation of individual resources and people will neither be trusted. It is important that the trust in an online community should be made explicit, for example by showing the community’s output. Relevance & Quality: The quality of an individual contribution and contributor relates with the way the community uses this contribution. Scalability: “A one-on-one system in which a trusted professor manually certifies a student does not scale well. A community-based reputation or voting system scales more easily, but outsiders might question its reliability.” Human online interactions with content can be used to calculate and predict things like expertise and value, but these calculations must be trusted in order to have value. Trust in the system as a whole is a crucial requirement. Transparency: Different assessment methods must be in place to reduce bias. These methods, involving both human and digital resources, must be transparent, because open systems encourage inspection and improvements and lead to higher standards of accountability.
What we can expect in an open system of assessment is that achievement will be in some way 'recognized' by a community. This removes assessment from the hands of 'experts' who continue to 'measure' achievement. And it places assessment into the hands of the wider community. Individuals will be accorded credentials only when people in the community think they deserve to be accredited. Downes describes two mechanisms of such a system: • • A mechanism whereby a person's accomplishments may be displayed and observed. A mechanism that constitutes the actual recognition of those accomplishments.
2.2.2 Sustainability of online communities The above descriptions argue that self-organization and peer-based learning in online communities can become an important and effective mode for learning. Sustainability of selforganizing online communities is a complex issue, that relates to a number of social and technical factors. Supporting people to create new communities has the potential to improve communication and support sharing of critical information and knowledge. It also aligns with newer organizational views: moving from command and control to more competency-based virtual communities . Despite the popularity of online social networking sites, most initiatives fail to reach momentum and fade away shortly after inception, because individuals lack the motivation to be active [128,129,130]. In many online communities and websites that rely on community participation, the majority of the contributions are done by a very small percentage of members.
Understanding of the factors that influence motivation of individuals in online knowledge environments can be used to increase willingness to invest time and share knowledge. The following sections summarize mechanisms that influence motivation of individuals in online knowledge environments, and their willingness to invest time and share knowledge. There has been a substantial amount of research on motivation to share knowledge and increase participation in online communities. There are various descriptive and prescriptive frameworks that relate to the design of social or community software [107,132], community sustainability , and drivers of community participation and contribution [25,65,132,134,135]. Many studies point out that social and technical factors need attention in order to enable knowledge sharing [132,134,136,137,138]. Thus, sustainability of peer-based learning community contains elements of technology and organization. Reputation Reputation relates to the concept of online identity and trust and is a primary research focus in Web science2. Overview of past actions and participant identification helps to create and sustain trust relationships in communities . Trust forms the basis of a relationship and is one of the most important enablers of community participation  and sharing knowledge . Reputation can be used as a virtual currency and is a conduit for trust in online environments. Reputation is also used as a basis for recommendations and connections between people and content. Rheingold (1993) in his discussion of the WELL, lists the desire for status or prestige as a key motivation of individuals' contributions to the group . Especially in knowledge or information sharing communities, recognition, status and reputation form an important driver for individuals to contribute [66,140]. In Maslow’s hierarchy of needs, helping others and being recognized because of that satisfies a person’s need for esteem . Wasko and others apply theories of collective action to examine how individual motivations and social capital influence knowledge contribution in electronic network. They conclude that people contribute their knowledge when they perceive that it enhances their professional reputations[64,141]. Increased recognition also supports identity building and belonging . In online communities, people tend to share more information, when their contributions are visible to the community as a whole, and to the extent there is some recognition or praise or encouragement . Many communities have therefore features that show the level of contribution of individuals in ranking or increased visibility. In Open Source communities, programmers are motivated not only by intrinsic aspects, i.e. engaging in an activity out of pure pleasure, but also have in mind the signaling knowledge to potential employers of profit-oriented companies . The motivation of managers in OSS projects, as well as of programmers can be traced back to career plans, which makes the reputation one has within a community so essential . 2.2.3 Conclusions on sustainability As we can see, sustainability is a complex issue that is a goal in itself, but never an end. Sustainability requires continuous adaptation to be able to meet the requirements of an ever-
changing environment. The Internet is a great place for certain kinds of learning, but challenges relating with sustainability and quality are paramount. Online communities are complex social and technological environments, and the design of sustainable online communities very difficult. Sustaining peer-based learning in online communities concerns, amongst other things, (i) the production of useful learning resources, (ii) quality assurance, (iii) assessment and feedback, and (iv) recognition for learning activities (including those activities that really prove certain skills, such as explaining stuff to other people). We have seen various initiatives that are able to sustain the creation and sharing of highquality open educational resources, and some do even offer learning support. With regard to quality assurance in open education, most experts agree that a centralized approach toward quality assurance is less sustainable than a decentralized approach. Similarly, assessment and accreditation of learning achievements and activities cannot be done in a sustainable way if it is directed in a top-down structure. Siemens and Downes have shown that even in an online course with 2200 participants, feedback and guidance could be offered by the network, rather than the course managers . A more decentralized approach for assessment involves providing tools for peer-based feedback (ratings, comments etc.) and communication. Effective peer-based social learning in online communities requires participation on many different levels and involves many activities. Even though some elements are lost in online interactions (compared with face-to-face interactions), effective online learning in online communities can be sustained when enough people participate, consuming various roles in the process. These roles include the aforementioned peer-feedback and assessment, but also organization roles, aggregation of content, creation, etc. Most of the processes that relate to these roles can be considered forms of learning. Even though “learning” itself is an important incentive for people to participate, more incentives are needed to sustain an online environment. Literature on social mechanisms and motivation in online communities show that reputation and recognition are crucial in motivating people to be engaged in the process and share or create knowledge. Therefore, reputation could be used as an instrument to incentivize behavior that sustains the various processes of peer-based learning in online communities, including production and aggregation of resources, peer assessment, and providing guidance. Reputation can be used as recognition for someone’s activities and contributions in an online community. We have identified a lack of systems and models that support official recognition (accreditation) in peer-based online learning communities. The next chapter will focus on reputation and quality assurance systems. It addresses some typical reputation systems, and concludes with a research framework for reputation.
2.3 Reputation and trust
In the previous chapter, we concluded with challenges for peer-based learning in online communities. The Internet provides with numerous opportunities for active, self-regulated and networked learning. It is a giant network of networks in which the people communicate. The architecture and intuitive tools that have been built (especially the recent years) allow for creation and sharing information in social and professional networks. The Internet has been used to disseminate educational resources for free, and there have been projects that offer support and guidance, as well as educational technology, for free, in addition to the resources. It seems that scalable and sustainable models are being developed for massive online courses by supporting students to reflect and assess contributions in a peer-based manner. We have identified a lack of models and systems that support recognition of learning activities in peerto-peer communities. In this chapter, we look at different approaches to learn lessons for the design of a valid system that supports recognition and improves assessment in online communities. The need for such systems, and for research in this direction, is made clear by Schmidt et al. (2009): “..despite improvements in methodology, assessment practices have a tendency to focus on easily quantifiable measurements rather than contextualized behaviors, dispositions, and attitudes. For our open education accreditation model, we are interested in retaining the goal in accreditation of accurately reflecting learning and skills to enable individuals and firms to negotiate employment arrangements efficiently. However, we also acknowledge that the skills needed in the 21st century are radically different from those tested and accredited in the past. Open education communities have certain unique characteristics that are ideally suited to the development and recognition of such new abilities in its individual members.” They further argue that digital portfolios, digital trails (what you leave behind on the web), and aggregations of individual opinions and ratings are used to improve relevancy in online learning environments. Reputation models that calculate trust can enhance and improve the accuracy these environment. Kollock (1997) has shown that reputation is one of the fundamental motivations of people to share and create knowledge in online communities . In this proposal, we agree with Schmidt et al (2009), that… “..a better understanding of indicators for knowledge and skills in open education communities is needed. Such indicators would consider processes and describe types of communication and interaction as well as behaviors within a community of learners.” 2.3.1 Why reputation? Tuomi (2007) describes that the social importance of formal educational certificates is now declining, as the capabilities, interests, and reputations of people can be directly evaluated using information and communication technologies. "Instead of asking whether a job candidate has a formal educational status, a potential employer can now review the candidate’s actual track record, blog postings, and possibly e-portfolios." In some expertise areas, such as computer programming, employment opportunities often depend on a track record that can be reconstructed by search engines and personal blogs. The 24
digital identities of persons now consist of their own representations of achievements and experiences, as well as reputations that accumulate through the comments of others. Formal educational certificates may be components in such digital representations of capabilities, but their relative importance will diminish . Labalme and Burton (2002) call this reputation capital, and argue that such values can be carried through systems . McLure-Wasko and Faraj (2005) isolated individual motivations and social capital considerations as main influencers on knowledge sharing. They had found that people tend to actively contribute to online communities when they perceive that this enhances their professional reputations . In an essay on the future of online learning (2008), Stephen Downes writes the following; “What will emerge for learning institutions, as for most other services, is a system of reputation management that is integrated into the search process. Recommender systems, as such systems are now called, will employ pattern-matching software to find resource providers for potential clients. (Herlocker, Konstan, Terveen, & Riedl, 2004) The software will draw information from a wide range of other services, including information about the institution that produced the resource. As we have seen, though, with search engine optimization (SEO) and other attempts to mislead reputation systems, there will continue to be a tension between the trust we put in such systems and the degree to which they can be infiltrated or corrupted. (Wu, Goel, & Davison, 2006) Reputation systems based on data that can’t be replicated or imitated will acquire the most trust, and these will most likely be based on verifiable identity and interactions within social networks.” Because reputation and trust are highly interrelated topics, we will define them below, before continuing this chapter. 2.3.2 Trust and reputation Trust is the confidence in or reliance on some quality or attribute of a person or thing, or the truth of a statement. In fact, trust is often used interchangeably with related words like credibility, confidence or reliability. Trust is the basis for interpersonal interaction and especially for cooperation in a social network. Merriam-Websters Collegiate Dictionary states that trust is “..the assured reliance on the character, ability, strength, or truth of someone or something.”3 Kinateder and Rothermel (2003), from an Artificial Intelligence perspective, define trust in an entity as the belief that under certain circumstances this entity will perform in a certain way . Another definition of trust commonly found is the one of Diego Gambetta “... trust (or, symmetrically, distrust) is a particular level of the subjective probability with which an agent assesses that another agent or group of agents will perform a particular action...”. Lik Mui (2002) is adapting this definition slightly in emphasizing the importance of expectation instead of working with probability: “a subjective expectation an agent has about another’s future behavior based on the history of their encounters”.
Reputation systems are used to increase trust in persons or objects. The scientific research in the area of computational mechanisms for trust and reputation in virtual societies is a recent discipline oriented to increase the reliability and performance of electronic communities . Usually, reputation is based on actions or achievements done by individuals or groups. Whereas trust in between agents can be defined as a subjective expectation an agent has about future behavior , reputation has more global characteristics. Why are reputation systems so important for fostering trust among strangers? Online reputation systems are developed to obtain and maintain measures of trust between people, and intend to offer incentives to behave in a certain way. When people interact with one another over time, the history of past interactions informs them about their abilities and dispositions. The expectation of reciprocity or retaliation in future interactions creates an incentive for good behavior. An expectation that people will consider one another’s pasts in future interactions constrains behavior in the present . Game theoretic approaches to reputation are predominant nowadays, resulting in systems that are likely to give good results in scenarios that concern simple interaction patterns between human beings, such as those in online marketplaces. However, when the complexity of the scenario increases, these models are not so good. They reduce trust and reputation simply to a probability or perceived risk in decision-making . This seems to be too restrictive in scenarios where the complexity of the agents in terms of social relations and interaction is high. Reputation cannot be understood as a "static attribute, rigidly codified as footprints of social hierarchy" . On the contrary, it has dynamic properties, because reputation attribution is a socio-cognitive mechanism that takes root in communication processes. Both the "reputed agent" and the "reputing agent" should be taken into consideration, and the context and relevant processes in which trust is established. 2.3.3 Context, transitivity, and cross‐community reputation An important point regarding trust is the fact that trust is not transitive . Just because X trusts Y and Y trusts Z, doesn't necessarily mean that X would trust Z. The problem is usually one of context. So X can’t be sure in which context Y trusts Z. For this, approaches have been suggested to incorporate context or categories into trust systems . To translate too rigidly or literally from the sociological and psychological domains can be difficult and would lead to high dimensionality. Therefore, a set of categories or contexts would need to be chosen, for which the trust between peers could be transitive for individual categories and could be applied inter-categorically where possible. Golbeck and Hendler have proposed the reputation inference algorithm, to be used with semantic web based social networks, founded on the Friend-Of-A-Friend (FOAF) vocabulary . The FOAF project defines a mechanism for describing people and who their connections are. We extend that ontology by adding binary trust relations (trusts and distrusts). Golbeck and Hendler focus on social networks, and provide an analysis of trust between people and how this trust could be inferred. In this proposal, we focus more on the inference of trust from object to the object’s author. Transitivity then means that someone’s trust in an object (i.e. a paper) is inferred by the author/creator of that object. Many other trust and reputation systems
intend to provide a ‘global’ characteristic [76,155,156], but in the importance of context requires a different approach: reputation not as a global but a local characteristic. A reputation in one online community is usually not reused in another community . For example, a seller on eBay cannot bring use his/her reputation to another online marketplace. The other way around, it is also impossible to ‘bring’ your reputation to eBay and replace or merge it with your eBay reputation. The trust people have in eBay’s reputation system, and in the company, may be affected if it becomes dependent on other systems, managed outside eBay. Yet, there are considerable advantages for communities and individuals to share reputation between communities. Resnick et al. argue that limited distribution of feedback decreases its effectiveness, because reputation (both the good and the bad) relates to only a single online arena. The main advantages of using cross community reputations (CCR) are (i) leverage of reputation data from multiple communities; (ii) producing more accurate recommendations; (iii) reputation accumulation: a user does not have to build a reputation from scratch; (iv) users are able to maintain (global or community-specific) offline reputation certificates; and (v) faster establishment of new virtual communities by importing reputation data from related communities . Also, the trade of reputations may lead to new opportunities for communities as a reputation provider. A reputation provider for experts in solar energy for example, may offer communities and their members to share reputations across a network of solar energy companies, researchers, and institutes. For this to happen, reputations need to be constructed in a way that allows for interoperability and synchronization between communities. Kinateder & Rothermel (2003) propose a directed graph of categories to support mapping of dependencies and relationships in different reputation systems . Related context increases the likeliness of having similar incentives and ranking between communities, improving compatibility and thus the possibility of reputation mapping. Berlanga et al. (2009) warn for privacy issues that will arise when participants’ online identity and reputation are transferred from one community to another . There are two preconditions for cross-community reputation systems: ontology mapping and trust relationships. First, a community should define and follow a reputation ontology. If the most important reputation mechanisms and mappings have been described, elements of different ontologies can be mapped and related. Secondly, there is a trust relationship between communities. Concerning CCR, trust can be defined as the extent to which one community relies on another community to provide reputation for members of both communities . As trust depends on context, more contextually similar communities are more likely to be able to share and agree upon ontologies. This is explained in more detail in the next chapter. In the following section, we will look at various systems that address trust and reputation. The choice of systems relates with the topics identified in the previous sections. Important elements include (i) the motivation toward people to behave in a certain (positive) way, i.e. to share knowledge; (ii) reputation as a conduit for trust in quality or future behavior; (iii) the relevance of reputation in a professional context; (iv) the indirect nature of reputation; (v) the contextual nature of reputation; and (vi) the typology or ontology of reputation.
The contextual nature of reputation means that trust in reputation is also contextual. The proposed research describes how reputation of someone in one online community, may be useful and trusted by relevant 3rd parties. The contextual element of reputation should therefore be included in a generic reputation system. The first example of the series concerns Google, which is not so much a reputation system as a trust system. The system is able to provide relevant search results based on an inherently simple algorithm that is specifically aimed at containing the dynamic nature of the Web. Other examples focus on other aspects that are considered important, such as having a defined ontology for reputation (eBay), the professional relevance of reputation (Guru), and the indirect nature of reputation (StackOverflow, research community). 2.3.4 Examples of trust and reputation systems Reputation systems have been around since the beginning of the web, and form a crucial element for trust in many online communities, such as eBay. In their new book “Creating Web Reputation Systems”, Farmer and Glass (2010) offer a framework to describe a reputation statement .
Figure 3 - Reputation framework (Farmer & Glass, 2010)
Farmer and Glass distinguish a source, a claim, and a target as the basic constituents of a reputation statement. In the following sections we will use this framework to describe different reputation systems. By doing that, we hope to identify characteristics, advantages and disadvantages of online systems that assess the quality of objects (including persons). Google PageRank ‐ Trust in online content Even though Google's search engine is not directly considered a trust or reputation system, it is significant in this respect. If we define trust as the confidence in something or someone to perform or deliver something, we can look at Google as a system that ensures the probability of a result to deliver the result (answer) on the question (query) asked. Google has been added
to this list because it has proven to be able to deliver highly relevant search results for free, facing a rapidly expanding and more dynamic Web, and even become highly profitable. Google’s PageRank system calculates relevance of a web resource by looking at its relation with other web resources. PageRank relies on the uniquely democratic nature of the web and uses its vast link structure as an indicator of an individual page's value [159,160]. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. Votes are weighed according to the PageRank of the voter. The image below makes clear how this works, in a simplified form: A’s link to B is assigned more weight than c’s link to B, because of differences in popularity (PageRank). Next to popularity Google analyzes contextual factors of webpages and their relations to allow semantic searching.
Figure 4 - Google PageRank
As we said earlier, finding concerns two things: searching and storing. On the ‘storing’ side, Google accumulates information about a webpage or URI (Uniform Resource Identifier: digital object) by looking at its relations with other URIs. The ‘votes’ are contextualized through interpretation algorithms that analyze the content and the topics affiliated with a URI. On the ‘searching’ side, in most cases, Google ‘knows’ about the background of the searcher, and thus being able to provide better results than a search engine that does not know anything about the person who does the search. Making an analogue to learning, we have seen the same: learning happens more effectively when the background, expertise and cultural elements of the learner are taken into consideration. Another important element concerns the way in which Google interprets ‘quality’, or in this case, relevance. By measuring static and dynamic relationships between URIs, Google is able to define a notion of relevance of a webpage and knows how to connect this notion to specific contexts. It does not rely on specialized committees who analyze and evaluate the relevance
of a webpage, but looks at ‘the behavior’ of the Web by looking at the relations between URIs. Hence, by looking at relationships between resources, it is possible to create a sustainable and scalable trust system to determine the relative value of objects. Google’s focus is on relevancy of websites, rather than people, but a similar approach can be taken toward people. A webpage is not the same as a person, but Google PageRank approach is very relevant to consider in the development of a reputation system. If we look at Google PageRank from the perspective of the Farmer & Glass reputation framework, we see pages voting for each other, and that each vote also contains weight and context. This makes it so powerful and instructive, we should look at how people use and rate (vote for) contributions within communities, and determine how this influences the value of contributions. As with Google’s PageRank, we should be able to determine the weight and context (which are interrelated) of each vote. eBay – Trust as a basis for transactions The biggest online retailer would probably succumb under its own success if sellers were not subject to the site’s reputation system. Sellers are evaluated by four criteria: “Item as described”, “Communication”, “Shipping time”, and “Handling charges”. This is part of eBay’s reputation ontology, which describes interactions and relations and (ranges of) values that determine the reputation of an individual seller. Clearly, this reputation system is mainly developed with the objective of guiding the behavior and increase trust on eBay. For sellers, other ways for a seller to improve their trustworthiness is to sell more items and being labeled as favourite seller by buyers. Even though the system works (eBay hosts millions of transactions per day), it has its critics. An experiment by Resnick et al. (2006) examines the value of eBay’s reputation system . It was found that only 0.6% of all the ratings provided by buyers and only 1.6% of all the ratings provided by sellers were negative, which seems too low to reflect reality. The possible explanation they provided for the positive rating bias was that positive ratings are a sort of exchange of courtesies, whereas negative ratings are avoided because of fear of retaliation from the other party.
Figure 5 - eBay reputation
The above example shows some interesting issues for reputation systems.
Direct, explicit feedback: As shown by Resnick et al. (2006), direct feedback can be ambiguous . Anonymity may solve this problem, but may on the other hand cause problems as well. Implicit feedback: Google has shown with its Pagerank system that it is possible to determine relevancy or quality based on implicit factors. eBay shows number of transactions, as one of these implicit parameters of trust. History: Next to figures showing feedback, the sales history of an eBay seller can be accessed, including all the remarks made by buyers. Years active: An eBay seller’s profile always discloses the number of years or months active on eBay, which also is a strong indicator of trustworthiness: you will have to start all over again when you misbehave. Reputation ontology: Describing the parameters that influence reputation can be very important, and generic, such as in this example. First of all, it is a powerful instrument to guide behavior on an online community. Secondly, explicitly stating it could result in standardization in an industry and allow for integration and exchange of reputation data.
Guru.com – Reputation to get a job Malone & Laubacher (1998) coined the term ‘e-lance economy’, meaning an economy largely based on temporary organizations of individuals that emerge and dissolve when business opportunities arise and disappear, and where IT serves to link individual nodes . An elancer, or electronically connected freelancer, is described as entrepreneur, independent contractor, freelancer, independent consultant, or contingent worker. This kind of selfemployment is becoming more mainstream, and provides a number of advantages. Being employed by several employers at the same time creates more security, and opens more windows of opportunity . Online marketplaces, including Guru.com follow this idea, and provide a place for individuals and organizations to find each other and employ or be employed. The mechanism is similar to eBay but concerns the ability of people to do a specific task. More than being a good seller, someone on these platforms can be an expert in something. Reputation relates with the money earned (and other implicit parameters), and direct feedback by employers. Freelancers can also provide a resume, portfolio of work, and do standardized tests to prove certain basic skills. Another way to make yourself visible and increase reputation is to answer questions on the forum. The image below shows how freelancers are presented in Guru, showing the most important trust parameters, including badges (community accreditation).
Figure 6 - Guru.com reputation
An initiative that is more focused on learning is called Myngle, which is a language-learning platform. On it, people around the world act as teachers as well as a learners. You learn what you need, and put it practice what you know as a teacher or apply your skills directly and earn money doing that. We can draw the following conclusions: 31
Responsibility: First of all, members of these communities behave responsible, because their earnings depend on the trust that is conveyed through their reputation profiles. Again we see that reputation is a powerful mechanism to guide behavior and to sustain a online social platform for professionals environment. Ontology: Secondly, as with eBay, transactions are evaluated by both parties, where feedback is given. A more extensive overview of the topics covered by the community is also made clear. Portfolio/History: The history of employment and a portfolio is shown, giving employers more insight into expertise and competencies. Expertise: On eBay you can become a trustworthy seller by always keeping your promises. eBay’s context is selling goods. On professional online communities, trust is also contextual. On Guru, freelancers can be hired for almost all disciplines, Myngle teachers come from all over the world. Information about expertise on platforms is disclosed by freelancers and teachers themselves and evaluated through employment. The image below shows that the contextual information of an interaction/transaction is specified. For example, if a teacher on Myngle offers lessons in Italian and Spanish, and she is a native Spanish speaker and can hardly speak Italian, then het expertise profile will, over time, show that as well. Standardized tests: Internal and external accreditation possibilities improve trust in the system.
StackOverflow community – Question Answering StackOverflow is an online community for people discussing IT-related issues. On an advanced forum, community members can ask questions and/or provide answers. The answers receive votes by community, and the answers with the highest amount of votes is chosen as the best answer. In addition to votes and answers, members can add tags to questions and answers. These tags play an important role in the reputation of users. Part of member profiles is dynamically updated with the tags of Q&A topics they participate in, as shown in the picture below.
Figure 7 - StackOverflow
After analyzing the mechanics of the community and its reputation system, the following interesting elements are added to the discussion: • Combine context and value: The system is able to combine the contextual factors (tags assigned to questions) with votes to make a dynamic representation of a person. A profile consists of static information of a person (added by him or herself) and of dynamic information that incorporates expertise. Hence, members that answer questions build up a reputation that includes context. From sharing to employment: On this site, members of the StackOverflow community can add their resume and improve their professional profile, but their contributions in the community and their reputation are visible as well to 3rd parties. These are potential employers, and is an important incentive for the members to answer questions, especially questions that prove their expertise. Even though this seems unusual, it is a standard procedure for software giant IBM to look at a person’s reputation in an open source communities when employing someone. Sustaining lifelong learning: StackOverflow and similar communities show that sharing information for free can lead to employment and improve professional reputation. Active and self-directed learning is possible and sustained in decentralized communities, if the right tools and mechanisms are in place. Reputation, which is crucial for the electronic freelancer, depends on contributions. It is an important incentive to collaborate in order to get business opportunities.
Academic publishing and reputation Despite criticisms on the Impact Factor it is still widely regarded as one of the most important quality measurement and assessment systems in the scientific community . The Impact Factor is a measure reflecting the average number of citations to articles published in science and social science journals. It is frequently used as a proxy for the relative importance of a journal within its field, with journals with higher impact factors deemed to be more important than those with lower ones. The Impact Factor is shown in relation with other elements in the simplified figure below.
Figure 8 - Academic publishing
Universities use the Impact Factor to assess the quality of research of its departments, and allocate funds based on that assessment. Hence, the influence of the Impact Factor should not be underestimated. The Impact Factor received a lot of criticism because of its focus on the journal in which a paper is published, rather than the number of citations it receives [165,166,167]. In reaction, several other initiatives have emerged, including the h-index, and PLoS (Public Library of Science) Article Level Metrics, focusing more on individual impact and citation level. Currently, debates in the blogosphere focus on how this information can be improved with a social impact level, by measuring popularity and usage . Despite the different approaches to academic value and relevance, we can extract some useful elements: • Weight of source: The Impact Factor of a journal influences the value of individual papers. This is similar to PageRank that also assigns extra weight to links (votes) that come from popular websites. Even though there are different approaches, we see that sources in a reputation statement can carry a weight. Gaming the system: Self-citation is mentioned as a way to game the system, and to increase the impact factor of a paper. This however, is not always the case . Reputation systems that do not address these issues are less valid and therefore less trusted. Google develops advanced analysis tools to determine whether or not a website is gaming its PageRank system, including looking at usage (time on site, number of clickthroughs, etc.). This is a way to use the Web community to manage quality, which is a sustainable and scalable way to do it.
2.3.5 Conclusions on trust and reputation The examples show us a range of approaches toward quality and reputation management in online communities. Google’s PageRank is a highly scalable system that addresses the dynamics of the web, but is less able to distinguish between online content and the producers of content (people): it does not analyze the relative value of a person to do a specific task or to 34
know the answer for a question. eBay has developed a rather simple but effective trust mechanism by looking at sales history and allowing buyers to reflect on the purchase. Resnick (2006) argues however, that the feedback is not always to be trusted, because the direct link between the feedback and the buyer causes fear of retaliation when giving negative scores . On online marketplaces, such as Guru and Myngle, trust is described in terms of employment history, feedback, and artifacts, such as a digital portfolio. StackOverflow, an online community for IT-related issues, successfully links question-answering to an online reputation that is accessed by relevant 3rd parties (employers in the IT sector). A reputation statement By aggregating the results, a framework to research reputation in online communities is created. A reputation statement, as follows from the above analysis, contains three dimensions: 1. Value, expressing the relative level of trust, appreciation, or quality (i.e. "good" > 0.8 out of 1.0) 2. Context, describing all the relevant context factors, agreed upon by community and relating to learning and (especially) teaching activities. The context is described in a community specific metadata standard. the behavior that supports sustainability (i.e. "quality of feedback", "Ergonomics and RSI") > in the case of peer-based learning in online communities, context will relate to the community domain and specific skills and competencies that can be expressed through knowledge sharing and peer support 3. Weight, which is especially relevant in case of assessment of knowledge or quality of highly specific information: expertise should be acknowledged. The weight has a dynamic relationship with the context: the context determines the weight of a source with a reputation statement. The framework is shown in the picture below.
Figure 9 - Reputation scenario
The above picture shows a reputation statement from a person to an object. This is not the only possible way in which ‘value’ or quality can be expressed in a reputation statement,
anything that can be identified on the Internet can be both a source object and a target object in a reputation statement. Other possibilities are from one object to another (i.e. a citation in one paper to another), or a direct recommendation from one person to another (i.e. LinkedIn recommendations). In the above picture, a person makes a reputation statement about an object, i.e. a rating of a document. • The person is identified with a user ID, and has a reputation profile, described in metadata. The source’s reputation profile can be used to influence the weight assigned to the reputation statement. One could argue that if a person is a world expert in a certain topic, he/she is better able to express an opinion (reputation statement) on that topic. The reputation statement has a certain type that can also influence the weight of it, i.e. a rating for a document is more explicit than a single visit by a person, and may therefore carry more weight. Weight is a calculated value, based on the metadata of the source object, the type of value statement, and the metadata of the target object. The target object is identified and contains metadata. When the target object is not a person, then the target object has one or more authors (persons), who will inherit information from the reputation statement.
An important distinction must be made between person and object. Even though both can be source as well as target object (as explained above), a person does not have an author. An object, which can be an activity or a document is always affiliated with one or more persons. When a target object receives a rating, the reputation of the object’s author(s) is automatically changed. The argumentation for this logical consequence is explained by Resnick and others, who claim that direct feedback may be ambiguous [151,161]. Another very important aspect in the picture is the reputation ontology. The reputation ontology describes all the relevant elements for reputation profiles, including the most relevant (knowledge sharing and learning) activities that should be monitored, and the types of reputation statements. The ontology is something that is inherent to a community and is a common agreement on how things work in a community. If ontologies of similar communities are described in a similar way, it may be possible to use reputation data across communities, improving its value. The next chapter will focus entirely on the semantics of reputation, and ways to include relevant semantics into a reputation profile. We have described the relevance of context in reputation, and Semantic Web technologies, with standardized formats to annotate data, tools to create, share, use and combine ontologies, could be used to standardize the way information about achievements and contributions in peer-based learning communities are analyzed and stored. The Semantic Web is the idea of having a URI (Uniform Resource Identifier) for (each) word in a document. A URI is a link to a shared definition of the word. These definitions are usually stored in specific (metadata) libraries or repositories, such as the Dublin Core. The Semantic Web allows for expression of different types of relationships between people, objects and concepts. By using standardized and shared ontologies expressing individuals, profiles, social connections, and content, it provides a way to connect people and objects in an interoperable and extensible way. 36
2.4 Semantic web
The previous chapter shows an overview of trust and reputation systems on the Internet. A variety of approaches have been developed, all semantically different from one another, even though some of the systems share common features. In this chapter, we describe how the Semantic Web can improve reputation systems by providing a standardized framework to describe and annotate information on the Internet. In computer science, semantics means 'unambiguous'. The Semantic Web thus means making the web less ambiguous by defining rules for interpretation . Reputation is a very ambiguous term. A semantic standard for reputation in peer-based learning communities will not make reputation less ambiguous, but allows communities to adopt their own definitions of reputation in a standardized way. This will make interoperability of reputation profiles easier and therefore more probable. The proposed research will provide a standard model to assess the expertise of people based on the quality of content and interactions in online peer-based learning communities. Such a model describes, in a very abstract way, all the elements and relations that constitute reputation statements and how these reputation statement influence reputation profiles. The actual ‘calculation’ of reputation can be based on parameters defined in a community- or domain specific ontology. For example, the semantic model may describe the abstract term "reputation statement”. In a specific domain or community, specific vocabularies can be used for these evidences. An academic community, for instance, may use an ontology describing “citations” as important reputation statement, and “accepted_by_journal” as another important one. The Public Library of Science has started an initiative that explicitly includes all kinds of metrics, not only the Impact Factor of a journal . A developed semantic model also define the all the source and target objects that are able to cite, be cited, accept, and be accepted. 2.4.1 What is the Semantic Web? The semantic web is a large-scale, decentralized endeavor with the objective of bringing logic and meaning to the web. Currently, most web sites and resources on the web are described in a way that they can be understood by humans, but not by other computers. Formal and explicit agreement on meaning of concepts and the explicit translation into computer-readable language helps in making relevant information useful for computers and applications, offering a wide range of opportunities for calculation of relevance and making recommendations [71,172,173,174,175]. The semantic web is the idea of having a URI (resource identifier: link) for each word in a document. A URI is a link to a shared definition of the word. These definitions are usually stored in specific (metadata) libraries or repositories . The semantic web is made up of assertions, such as the following • • Thieme Hennis is the author of this document; The author of this document likes pancakes.
Now you can make up that Thieme Hennis likes pancakes. The semantic web specifies ways of exposing these kinds of assertions on the Web, so that third parties can combine them to
discover things that are true but not specified directly. Using syllogisms can be useful, but should be considered with care, Clay Shirky argues (2003): "In the real world, we are usually operating with partial, inconclusive or context-sensitive information." His critique mainly focuses on the fact that human beings talk in generalities, and that the real complexity of the world cannot be modeled using semantic technologies. The semantic web proponents do not argue that this will be done. The objective is to make agreements on how to annotate data, and as Jim Hendler says: 'a little semantics goes a long way'. He means that adding only a little semantic annotation may increase the value of an application or document significantly . The semantic web enables more effective communication, because with standard definitions and rules for interpretation, tools can be build that improve searching and filtering information, matching content and people, and allow for more decentralized ways of working . In the proposed research, existing frameworks and ontologies will be used to create a semantic standard that models the way reputation statements flow through a peer-based learning community. Example: Semantic Search We see an interesting example of the potential of the Semantic Web when we compare the search engines Google and Wolfram Alpha. In Google, if you type “What is the weather in Amsterdam?”, it will show you a range of search results, including hotels and businesses in Amsterdam, and different websites with weather forecasts. When we type in the same question in Wolfram Alpha, we are not presented with a list of links we have to interpret ourselves to find the weather in Amsterdam, but it shows us a page containing the current temperature, humidity, and all other weather related metrics for today. It also shows the weather during the whole year, using graphs and various metrics, and suggests additional metrics. Wolfram Alpha is a semantic search engine, and knows exactly what “What is the weather in Amsterdam” means. By using resources on the web that are also semantically annotated, it can understand the question, and find the relevant information to present the answer you are looking for. This is only possible when resources are semantically annotated. It has to be noted that Google is embracing semantic web technologies, as do other numerous other big and small companies, governments, and institutes around the globe.  2.4.2 Semantic Web Technologies The core of the Semantic Web concerns the following technologies; • • Content on the web is annotated with custom XML tags, like this: <person>Thieme Hennis</person> XML provides a syntax to encode data; the resource description framework is a mechanism to tell something about data. As its name indicates, it is not a language but a model for representing data about “things on the Web”. This type of data about data is called metadata. The “things” are resources in RDF vocabulary.
Relations are annotated in RDF triples; (Thieme Hennis)(is_engaged_to)(Sofie Blaisse). Triples are
Ontologies describe the relations between concepts and define the concepts, including the hierarchy; (person) has (father) & (mother). This is usually done within a viewpoint, meaning that the ontology is useful within a certain context. Inference and reasoning rules describe logical relationships, such as "If (person) is married, then (person) has (parents in law)". 2.4.3 Relevance of the Semantic Web So why is the Semantic Web an important domain that must be integrated in this research? Except for the most fundamental and common values (Thou shalt not..), in most other cases quality and expertise are concepts that have value only within a specified context. In the proposed research, we look at interaction patterns and how quality or expertise can be derived from these. Online behavior of people can be monitored and analyzed to provide a very rich picture of the community and its members and resources. The analysis of the behavior and patterns leads to profiles of people, resources, and communities as a whole. A standard for analyzing and storing reputation information allows for exchange between communities, and so data from different sources can be leveraged to improve trust and quality of recommendations. Another advantage is the expansion of the network of resources and people, being able to find relevant people and resources across networks and communities, rather than within communities. In a growing knowledge economy, applications and tools that enable searching and finding relevant persons across the Web will have high impact and commercial value. Semantic Web technologies improve both sides of the search-coin: storing and retrieving information from large data sources. Because relations are explicitly stated, and concepts are defined in unambiguous ways (if possible), interesting combinations and calculations can be made. One of the imagined applications may be a system that is capable of describing the value of someone based on activities online. If these activities are not explicitly described, the context in which value emerges, cannot be attained by computers. Therefore, semantic annotation of relevant concepts will improve the final artifact in terms of quality of information stored in reputation profiles, trust through transparency, and ability to store contextual information in a decentralized way. Also, Semantic Web technologies support the creation of ontologies and other artifacts.
3 Scientific approach: philosophy, strategy, and planning
Richard de Neufville describes four issues that need to be addressed in the formulation of a research project : 1. 2. 3. 4. What is the question or issue? What method will be used to address these questions? What evidence can be applied? What logic integrates the above?
Based on the problem statement, and the issues identified in the literature review, the following section will describe the main research question and a number of operational research questions. It further elaborates on the research philosophy, strategy and the required research instruments. The planning of the research is also presented.
3.1 Research questions
A thesis is helpfully framed as a question that defines the issue under consideration, comparable with a hypothesis. Research questions must have more than one possible answer. If a research question cannot be refuted it is a truism of no real interest . Several researchers have shown the importance of reputation as a motivation to share knowledge in online communities. The objective in the long run is to improve sustainability and self-organization in peer-based learning communities, by means of an incentive that continuously motivates people to behave according community standards. The core topic of this research is thus reputation and its relation with learning activities in online communities. The main research question is aimed at the design and evaluation of a reputation system to provide a continuous incentive for individuals to share knowledge and engage in teaching and learning support activities in online communities. On the one hand, reputation systems must be designed such that it guides behavior in a way that it sustains and supports the most important activities in a community. In learning communities, these consist of knowledge creation and sharing, learning support and guidance, and feedback and assessment. The nature of these activities provides us with an opportunity not only to model a community ‘value’ or trust, but specific expertise and competencies. Expertise and competencies are expressed through a published paper, providing useful feedback, or some kind of learning support or guiding activity. The content that is produced, interactions in which knowledge is shared, and peer-guidance activities and teaching are all things that can be monitored and analyzed to create expertise profiles of individuals. The proposed research system, therefore, must be able to recognize relevant activities and reputation statements, and develop reputation profiles that can be used to determine someone’s expertise or knowledge. The main research question is:
How does a scalable and trusted online reputation system look like that creates expert profiles from learning support, feedback, teaching, and knowledge-sharing activities in a peer-based, networked learning environment?
Scalability is important, because that is one element we have identified in the problem statement: traditional approaches to assessment and accreditation (a form of reputation) of individuals is not scalable to the online context; only a very limited amount of people can be assessed for credit. Trust is another essential element, and concerns the fact that reputation, when trusted across communities and especially by 3rd parties (including potential employers), becomes a strong motivation. Finally, the reputation system must ensure that activities are monitored that sustain and support peer-based learning. In this research, the nature of these activities allow for expert profiling. 3.1.1 Operational research questions The sub-research questions focus on these four elements: activities that sustain peer-based learning, the modeling of expertise or value based on these activities, scalability, and trust in the reputation profiles. Defining peer‐based learning First, we have to define peer-based learning in online communities, and develop an ontology describing useful elements, relations, and concepts that, as a whole, constitute knowledge sharing and learning support. The research is less focused on motivating learners to consume information than motivating people to share knowledge (to learn from) and to offer support to people who want to learn. • How can knowledge sharing and learning-support in peer-based online communities be characterized?
Inference of knowledge Next, we must analyze the relation between knowledge sharing activities and learning-support and knowledge and competencies. We have seen that inference of attributes can be difficult, if context is not addressed. • How can individual knowledge and competencies be inferred from shared knowledge and learning support activities?
Scalability The third requirement concerns scalability. Scalability in this research means the ability of the system to produce accurate reputation profiles with increasing numbers of users. Another dimension of the scalability is its adoption rate: can the reputation system easily be adopted by learning communities and remain useful? As we indicated earlier, the Semantic Web and its technologies are a basis to work from. • How can Semantic Web technologies increase scalability of the reputation system?
Trust Finally, the produced reputation may be scalable, but if it is not trusted, then it remains useless. Trust has two elements: (i) trust by the reputed agent that personal information is not shared with 3rd parties without consent; and (ii) trust by 3rd parties in the produced reputations. The first concerns transparent and explicit rules about ownership and control. The latter is merely concerned with the quality of the reputation profiles. The two perspectives do not have opposing concerns, but there may be a tender balance between privacy and control on the one hand, and quality of the profiles on the other. • How can trust in the reputation system be ensured from the perspective of the reputed agent as well as the interested 3rd party?
The objective of the research is to create a system that can be used to communicate the value an individual brings to a learning community. Because learning is a social activity, in which meaning is constructed, learning activities could include blogging, question answering, editing a wiki page, and publishing content (including interactive media). It is not the intention of the research to describe in detail all these activities, because they may be different for each community. What is the intention though, is creating a generic system that is able to adopt specific descriptions and ontologies of peer-based, online learning. The system must be flexible enough to be tuned to specific requirements set by one community, meanwhile adhering to the same standard as other communities to make reputation shareable and trusted in a wider context. The product of the research constitutes of a reputation system that is defined according to semantic web standards, a methodology to design reputation systems for peer-based learning communities, and a report. Below, the scientific and societal relevance are defined.
3.2 Societal and scientific relevance
This year’s Pew Internet survey on the impact of the Internet on institutions in the future describes the following: "The respondents who addressed the issue of “innovative forms of online cooperation” sometimes referred to activities between people and institutions that were post-bureaucratic. They argued that people could use the Internet and cell phones to create alternative, unbureaucratic structures to solve problems through network-structured communities." One of the important characteristics of these innovative ways of learning and collaboration is that it is networked: you should be able to find the relevant person in various online networks. Repeating the earlier statement that finding involves storing and searching, we can argue that, if online communities and professional networks maintain their own standard for the storage of user profiles, this vision is not likely to become truth. Standardization and interoperability improves the Internet as a place for collaborative, networked learning. A standardized and interoperable reputation system improves the sustainability of peer-based learning communities. This increases access to all elements of education, even professional guidance, to anyone with Internet access. It also supports electronic freelancers in profiling
themselves, and innovative and networked structures (with reputation/identity providers) for work may emerge from the learning communities. Not only online communities may benefit from such a standard, but traditional educational institutes as well. By adopting the standard, they might be able to decentralize a part of their efforts in education, and focus on specific activities that cannot be done online. They may even partner with trusted online communities in providing accreditation for reputation, or engage in other innovative collaborations. Finally, in a broad sense, the reputation system may be applicable in a much wider context than only the peer-based learning communities. In organizations, for example, the same reasoning may be applied to find experts, and provide an incentive for the people to share knowledge or comment on shared information. Specific scientific contributions are the improvement of ontologies, algorithms and technologies that relate with expert modeling. For education science, this research may provide with a useful model and technology to support for peer-assessment and measure and describe knowledge and competencies.
3.3 Research philosophy and methodology
Research philosophy defines a view on reality and a view on knowledge giving us the ability to choose an appropriate method to conduct meaningful and valid research. Research philosophy is built on an ontological approach (the way the nature of reality is viewed), a theory of knowledge acquisition (epistemology), and the practical approach to knowledge acquisition (methodology). Epistemology and methodology are intimately related: the former involves the philosophy of how we come to know the world and the latter involves the practice. Validity is guaranteed through rigorous and correct application of the research methodology. Mainly based on the ontology, epistemology and methodology we can differentiate among different paradigms. A paradigm in research philosophy defines what falls within and outside the limits of legitimate inquiry. It also states that different assumptions concerning in paradigms “cannot be dismissed as mere “philosophical” differences; implicitly or explicitly, these positions have important consequences for the practical conduct of inquire” . In scientific literature, 4 paradigms are identified: positivism, post-positivism, critical theory and constructivism [181,182]. Positivism is historically the oldest paradigm dominating natural and social sciences for centuries. Positivism states that reality is driven by natural laws and mechanisms, which can be comprehended and studied. The inquirer should not influence his object of study and should therefore take the necessary measures to avoid this. As such he should be capable of formulating an almost exact representation (e.g. a set of definitions for the laws) of this reality. Post-positivism was born as a criticism on positivism. It tries to be less certain about claims about reality. It is still close related to positivism and states an objective reality but the apprehension of this reality can only be imperfect and incomplete. Reality can only be known approximately, thus without every fully know it. Critical theory functions as an umbrella paradigm for several alternative paradigms. The common assumption of all these paradigms is
value-depended nature of inquiry. Reality is generally viewed as something shaped by different values, e.g. economical and social. The inquirer is part of this reality and as such influences the inquiry by its own values. Constructivism is finally a paradigm stating all knowledge is constructed and depends on convention, human perception and social experience. The inquirer and his object of study are strongly linked as the findings are created as the inquiry proceeds. A way to understand these paradigms is to analyze them from the ontological, epistemological and methodological point of view. An overview is structured in the table below with a column for each paradigm and a row for each point of view.
Positivism Ontology Naïve realism, “real” reality but apprehendable Post-positivism Critical realism, “real” reality but only imperfectly and probabilistically apprehendable Critical Theory Historical realism, virtual reality shaped by social, political, cultural, economic, ethnic, and gender values, crystallized over time Transactional/ subjectivist, value mediated findings Constructivism Relativism, local and specific constructed realities
Dualist/ objectivist, findings true
Modified dualist/ objectivist, critical tradition/ community, findings probably true Modified experimental / manipulative, critical multiplism, falsification of hypotheses, may include qualitative methods
Transactional/ subjectivist, created findings
Experimental/ manipulative, verification of hypotheses, chiefly quantitative methods
Tabel 1 - A comparison of inquiry paradigms in terms of ontology, epistemology, and methodology 
Our research subject is in essence a ‘wicked problem’, as defined by Hevner et al. (2004) . That is, our problem is characterized by: • • • • Unstable requirements and constraints based upon ill-defined environmental contexts. Complex interactions among subcomponents of the problem and its solution. Inherent flexibility to change design processes as well as design artifacts. A critical dependence upon human cognitive abilities to produce effective solutions. 45
A critical dependence upon human social abilities to produce effective solutions.
This leads us to frame our research as both inductive and explorative. We are explicitly not testing existing theories on learning or trust, although we are using theories to improve developed artifacts. What we aim to do in this research is build theory and two artifacts (a reputation system and a semantic model) that enable new forms of collaboration and learning. We approach the research through inductive reasoning, which begins with specific observations and measures in order to detect patterns and regularities, formulate tentative hypotheses that can be explored, and finally end up developing a general conclusion or theory. Through the analysis of several online communities, some generic characteristics can be formulated, and described in an ontology for peer-based learning in online communities. This ontology forms the input for the reputation system. 3.3.1 Ontology and epistemology Flood, in his synthesis of critical social theory and systems thinking, describes five ontoepistemological views and their primary concerns [184,185]. He describes five ontoepistemological viewpoints, capturing the subjective and objective extremes of ontology (realist versus nominalist) and epistemology (positivist versus anti-positivist). As we discussed above, the research aims to develop theory and create two artifacts (a reputation system and a semantic model). These artifacts are created through the use of a clear understanding of the relation between learning theories, reputation, and motivation, using an accepted classification framework for the description of relations and objects on the Internet. As the research focuses on a social-technical problem, realist ontology is appropriate. Realism is the theory that a reality exists outside of us, and independent observers therefore can agree upon that reality. Post-positivism states that facts are fluid and elusive, forcing us to only focus on our observational claims. Reality can only be known approximately; we cannot ever fully know it. Thus, validity of knowledge of social reality is of low resolution, but with a view to working on increasing validity. We therefore argue that we can position our research within Flood’s “Distant Ontologist” (Type II) onto-epistemological viewpoint, which means that validity and methodology are our main concerns for our research approach. Facts about reality can be derived from the investigation, and can be used to develop a theory and artifacts. 3.3.2 Methodology The research philosophy guides us to the research methodology. Methodology is the practical approach for knowledge acquisition. Methodology is focused on the specific ways – the methods – that can be used to increase understanding about the world. There are two recognized methods: the quantitative method and the qualitative method. Qualitative methods include the case study, phenomenology, grounded theory, and ethnography, among others. Quantitative methods include hypothesis testing, power analysis, met analysis, observational studies, re sampling, randomized controlled trials, regression analysis, multilevel modeling, and high-dimensional data analysis, among others. The proposed research is aimed at
developing a semantic model as well as a reputation system. This inclines that qualitative methods are in place to improve understanding about what constitutes reputation in the defined context, and to develop the semantic model and reputation system. Quantitative methods can then be used to test the scalability and usefulness of the system with large numbers of users. Problem‐solving process Taking the notions of ‘wicked problems’ and of bounded rationality as a starting-point, we will describe the process of problem solving, based upon work by Mitroff (1974), and presented by Sol (1982) [186,187]. Mitroff’s process consists of six activities and four stages. The cycle starts with the problem as being perceived in reality; e.g. by a problem owner. Through a task of conceptualization a conceptual model is constructed, which defines the variables that will be used to specify the nature of the problem in broad terms. This conceptual model, according to Sol, is formulated through the choice of a vehicle of communication, a worldview, and a construct paradigm. The empirical model specifies the conceptual model in terms of the system under study, and can be checked against the problem as it was originally perceived; the correspondence check. With the empirical model solutions can be generated and analyzed. Solutions must be checked on consistency with respect to the conceptual model. After the choice of one solution, it will be implemented in the real world, which can form the beginning of a new cycle.
Figure 10 - The problem-solving process (based on Mitroff 1974) 
3.3.3 Summary of research philosophy In summary, we set out to do inductive, explorative research. We have realism as our ontology, with post-positivism as our epistemology, and the qualitative method as our methodology. The proposed research, with the objective of designing a semantic model and 47
reputation system, leads to design science as our research approach. In the design science paradigm knowledge and understanding of a problem domain and its solution are achieved in the building and application of the designed artifact [183,188]. We argue that design science is a good fit for Flood’s Distant Ontologist (Type II), as design science’s focus on rigor and relevance fulfill the Distant Ontologist‘s need for validity and correct methodology.
3.4 Methodology: Design Science
In the broadest terms there has been consensus: that design science research in information systems involves learning through the act of building . According to Hevner, March et al (2004), two paradigms characterize much of the research in the Information Systems discipline: behavioral science and design science . The behavioral science paradigm seeks to develop and verify theories that explain or predict human or organizational behavior. The design-science paradigm seeks to extend the boundaries of human and organizational capabilities by creating new and innovative artifacts. As Hevner, March et al consider both paradigms foundational to the IS discipline, they use both paradigms to describe the performance of design-science research in Information Systems via a concise conceptual framework and clear guidelines for understanding, executing, and evaluating the research. The design-science paradigm has its roots in engineering and the sciences of the artificial . It is fundamentally a problem-solving paradigm. It seeks to create innovations that define the ideas, practices, technical capabilities, and products through which the analysis, design, implementation, and use of information systems can be effectively and efficiently accomplished.  Hevner, March et al consider designing useful artifacts complex, due to the need for creative advances in domain areas in which existing theory is often insufficient. They broadly define IT artifacts as constructs (vocabulary and symbols), models (abstractions and representations), methods (algorithms and practices), and instantiations (implemented and prototype systems). Design science creates and evaluates IT artifacts intended to solve identified organizational problems.’ These are assessed through evaluation methods that are scientifically valid and create the opportunity to refine the aforementioned theories and methods. Rigor is derived from the effective use of the knowledge base – theoretical foundations and research methodologies. To not loose sight on the applicability of the research in real world environments, relevance is achieved by conducting research that addresses business needs. The output of the research is material useable in these environments. Success is predicated on the researcher’s skilled selection of appropriate techniques to develop or construct a theory or artifact and the selection of appropriate means to justify the theory or evaluate the artifact.  The further evaluation of a new artifact in a given organizational context affords the opportunity to apply empirical and qualitative methods. The rich phenomena that emerge from the interaction of people, organizations, and technology may need to be qualitatively assessed to yield an understanding of the phenomena adequate for theory development or problem solving. 
The design process, used in design science, is a sequence of expert activities that produces the design artifact, as depicted in figure below. The evaluation of the artifact then provides feedback information and a better understanding of the problem in order to improve both the quality of the product and the design process.  IS research is conducted in two complementary phases. Behavioral science addresses research through the development and justification of theories that explain or predict phenomena related to the identified business need. Design science addresses research through the building and evaluation of artifacts designed to meet the identified business need. The goal of behavioral- science research is truth. The goal of design-science research is utility.  Hevner describes three similar cycles that make Design Science research rigorous and relevant, see picture below. These are the Design cycle, the Relevancy cycle, and the Rigor cycle. It is not uncommon for a researcher to address all cycles at the same time.
Figure 11 - Information Systems Research Framework 
Design science is especially suitable for the development of ‘supporting tools’; e.g. methods, tools, approaches that can be implemented in organizations or systems to improve the ways people are working. When complex solutions have to be designed rigorously, while remaining relevant for stakeholders, design science offers a suitable framework for research . 3.4.1 Guidelines Design science is inherently a problem solving process. To help in this process, Hevner, March et al (2004) have derived seven guidelines, based on the fundamental principle of
design-science research: that knowledge and understanding of a design problem and its solution are acquired in the building and application of an artifact.
Guideline 1. Design as an artifact 2. Problem relevance Description Design-science research must produce a viable artifact in the form of a construct, a model, a method, or an instantiation. The objective of design-science research is to develop technology-based solutions to important and relevant business problems. The utility, quality, and efficacy of a design artifact must be rigorously demonstrated via well-executed evaluation methods. Effective design-science research must provide clear and verifiable contributions in the areas of the design artifact, design foundations, and / or design methodologies. Design-science research relies upon the application of rigorous methods in both the construction and evaluation of the design artifact. The search for an effective artifact requires utilizing available means to reach desired ends while satisfying laws in the problem environment. Design-science research must be presented effectively both to technology-oriented as well as management-oriented audiences.
3. Design evaluation 4. Research contributions
5. Research rigor
6. Design as a search process
7. Communication of the research
Tabel 2 - Design Science Research guidelines 
3.4.2 Design Science activities Artifacts constructed in Design Science research are rarely full-grown information systems that are used in practice. More often, artifacts are innovative concepts that define the ideas, practices, technical capabilities, and products through which the analysis, design, implementation, and use of information systems can be accomplished. Kuechler and Vaishnavi (2008) regard design theory as a prescriptive statement, a significant, possibly the most significant, output of the research effort . Venable (2006) created a framework for Design Science, describing design science activities and theory development , based on Kuechler and Vaishnavi (2008). For our research, we adopt this framework, which is shown below, as the outlook on design science as a research strategy.
Figure 12 - Design Science activity framework 
3.5 Design cycles and instruments
Our chosen research strategy provides us with the steps that need to be taken in our research, and also the research instruments that are needed to support our research strategy. Following both Sol’s process of problem solving (Figure 10), and Venable’s activity framework (Figure 12), we will use three of the Venable’s design science activities (Problem Diagnosis, Technology Invention / Design, Technology Evaluation) in a sequential design cycle. All these activities actively contribute to the fourth activity: Theory Building (e.g. problem theories and solutions). Our research planning will utilize two of these design cycles: one for developing the semantic model of reputation, and one for designing the reputation system. Thus, in our research we will start out with gaining an increased understanding of the problem, as perceived in reality. Based on this analysis, a conceptual model of our technology design will be made. This model will be evaluated, resulting in an empirical model. All these steps, and their results, contribute towards theory building and finding a solution for the original research problem. This approach translates to two research cycles, each comprised of three steps: problem diagnosis, technology invention/design, and technology evaluation. 3.5.1 First cycle: developing the semantic model The first cycle is aimed at developing the semantic representation of reputation in peer-based learning communities. This model of reputation intends to improve motivation, by making a representation of someone’s value or expertise, based on knowledge-sharing, learning
support, and teaching activities. The model adheres to semantic web standards, for reasons explained in chapter 2.4. The specific steps are defined below: Problem diagnosis In this step, we have to describe typical activities of knowledge-sharing, learning support, and teaching activities in online communities. Relevant activities are activities in which knowledge is created or shared, evaluated, commented on, and happen in a peer-2-peer manner. • • Literature on active and self-organized learning will form the initial input for describing the generic characteristics of the learning, and create an ontology for this type of learning. In addition, the online community Peer 2 Peer University4 will be analyzed using social network analysis and data mining in order to derive generic elements for the conceptual model.
With regards to theory building, this research step results in an overview of incentives for offering peer-support, and a description of typical activities that sustain learning in a peerbased online community. Technology invention / design The design phase is aimed at translating the conceptual model of peer-based learning and teaching, and the relevant aspects that incentivize behavior, into a semantically correct model, abstract enough to be implemented in various communities, and flexible enough to adopt existing community-specific ontologies. Adopting the Web Ontology Language, the Resource Description Framework, and other semantic standards, a semantic representation will be made of the conceptual model. In order to improve acceptance and prevent redundance, if possible, existing elements are used. For instance, the model could be developed as an extension of the widely adopted FOAF (Friend-of-a-Friend)5 standard. Evaluation The developed semantic model of reputation (as defined in this research) will be shared online, using my personal wiki6. Next to making every step visible and available, important updates and relevant questions will be pushed to experts in the field, and evaluated through interviews. Everything will be shared with experts in the field, and worked on during at least one so-called VOCamp (Vocabulary Camp)7. During these events, vocabularies and ontologies are created and improved in interactive development sessions with a small number of experts. The fit and relevance of the model is analyzed in two case studies. • First, through application in the online peer-based learning community “Peer 2 Peer University”, the applicability is tested. This case study will involve interviews with the
http://p2pu.org/ http://www.foaf-project.org/ 6 http://wiki.myopen.org/ 7 http://vocamp.org/
main developers of the platform, and will evaluate the usefulness of individual elements and relations described in the model. Secondly, applying it in the social academic environment PLoS One8, in Article Level Metrics may be used to ‘prove’ expertise and a contextualized notion of value of a paper and its author(s).
3.5.2 Second cycle: developing the reputation system The second cycle is aimed at developing the reputation system. The high ambition level of the research requires us to focus on the most relevant topics. This cycle will mainly be focused on developing and improving the algorithm that is able to analyze, interpret, and contextualize reputation statements according to the descriptions and relations in the semantic model. There are several requirements for this system. As we described in chapter 2.3, the transitivity of trust (and thus reputation) depends heavily on context. The semantic model of reputation does not only include the typical value statements, but also describes interoperability with community-specific ontologies on reputation (statements). The developed reputation system must therefore be able to combine the context of a reputation statement with the value expressed. Another requirement is the inclusion of weight into reputation statements. It is not in the focus of the research to make a user-friendly interface, or reputation management program. Rather than the technical implementation of the algorithm, we look at societal implementation and implications, and describe requirements that relate with privacy and identity provision. Problem diagnosis A system that is able to derive a notion of value or expertise from interactions and activities by actors in social online environment is very useful in learning environments and organizations where knowledge transfer and localization is fundamental to the business. In the problem diagnosis, we will focus on identifying elements in an online reputation that will provide the highest value for both single users as well as whole communities. This includes the following. • An extensive literature analysis on trust and reputation will result in an overview of existing trust and reputation algorithms, with their merits and deficits. We will match the requirements described in the semantic model of the previous cycle to determine the various approaches and develop a new approach. In-depth interviews amongst human resource managers and potential employers on StackOverflow9 will shed light on the requirements on the reputation system regarding trust in such a system. Interviews with members of the Peer 2 Peer University will provide us with insight in user requirements concerned with privacy and personal information as well as online representation and ownership of information.
Regarding theory building, this part of the research will advance knowledge on trust and reputation system by (i) proposing a novel approach for integrating context into a value statement; and (ii) explicating user requirements on trust and reputation that carries information about expertise and knowledge. Technology invention / design Using the outcomes of the case study analysis and interviews and literature review on trust and reputation systems, an algorithm will be constructed that is able to model reputation based on knowledge-sharing and learning support activities for any p2p learning context. It includes a way to make the reputation transitive and increase external validation and usefulness outside the boundaries of a single community. Using ProLog10, a popular logic programming language used for natural language processing, theorem proving, expert systems, etc., the algorithm is translated into a reputation system. Based on the user needs, an algorithm for reputation will be constructed and deployed in the Peer 2 Peer University. The reputation system accommodates for different contexts with an ability to use specialized ontologies, improve external validity and trust in the system. Theory building is focused on showing the relevance of this approach in comparison with other existing approaches. Also, a method is developed that makes it easier to implement the algorithm in an existing community. Technology evaluation The final implementation and evaluation of the algorithm happens through different experiments on Peer 2 Peer University and other peer-based learning places online. • First, using available data from PLoS One, StackOverflow, and Peer 2 Peer University, different deployments of the algorithm can be evaluated. Does the algorithm improve existing user profiles by providing insight in value added to the community? This means that user profiles are compared, and by means of interviews, evaluated with the end-users. Secondly, a course on Peer 2 Peer University, dealing with this subject, is started. In line with the objective of the research, learning and teaching happen (at least partially) peerbased. In order to test the resilience of the system, some of the participants will be asked to game the system (with the objective to improve their own reputation). Others will keep an eye on irregular behavior, just as what would happen normally. Thirdly, by means of a Delphi study, the scalability of the system is tested. Scaling can be interpreted in terms of number of users, but also the numbers of communities being able to adopt and deploy the system. The fourth evaluation criteria is concerned with the external validity of the reputation profiles: are they in fact useful for potential employers or other people outside of the community in which the profile is created? Through a questionnaire amongst reputed agents as well as potential employers, requirements and circumstances for trust in reputation profiles will emerge. This can be used for further research and development.
Finally, a comparative analysis will be done using Social Network Analysis. The activities and interactions measured on the Peer 2 Peer University at the beginning of the research will be compared with interactions at the end of the research.
3.5.3 Research instruments Thus, through our research steps, the following instruments are used: case study research, social network analysis, interviews, expert survey, and literature review. Case study research The essence of a case study, the central tendency among all types of case study, is that it tries to illuminate a decision or set of decisions: why they were taken, how they were implemented, and with what result. Thus, a case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident . Case studies will be used for evaluating and redesigning both the semantic model and the algorithm/reputation system. For the case studies, we will use the action research approach, in which the researcher is an integral part of the phenomenon and whose input influences the outcomes . A case study will be designed, executed and reported on for descriptive purposes (finding data and instances in which to evaluate and validate our design artifacts). A report will be written about the case study findings. With case studies, we run the risk of difficulties with generalizing or different interpretations; some arguments can be used to justify actual generalization of a concept or requirement, but mainly it is a choice one makes in the path from generalization (and less insight) to specialization (and more insight). A major challenge will be to find an adequate problem situation or inquiry setting. Many limitations, with regards to availability, credibility, location, suitability, openness and confidentiality affect not only the actual getting of a case but also the possibility of successfully gaining insight from it. The case studies in our research will be used to test the design artifacts. The goal of the case studies is to see which parts of the design artifacts work and don't work, with the purpose of learning from mistakes, and adapting the artifacts accordingly. Relevant case studies are those that share relevant data, or offer some way of support within the context of the research. An essential element must be that reputation could be used as a way to sustain activities and learning. Ideally, the case studies come from corporations and governmental organizations, as this will provide us with cases that are both rooted in reality and practicality, and have an actual need behind them. The following cases are proposed, but along the research, other cases may be used. Case 1: Peer 2 Peer University (P2PU.org) is an online community of open study groups for short university-level courses. The vision of the founders is similar to what is proposed in this research. It is also a research project, and the founders have expressed their need for the system proposed in this research.
Case 2: StackOverflow is a question answering community, where problems are solved in a highly decentralized way. The reason for including this initiative is the availability of data combined with the highly sophisticated reputation system. Reputation in this system is a very important incentive, and reputation profiles are used as additions to online curricula on the same site (they also provide a framework for jobs). PLoS One: The Public Library of Science an international, peer-reviewed, open-access, online publication. PLoS ONE welcomes reports on primary research from any scientific discipline. Especially relevant for this research is their 2009 initiative, called “Article-level Metrics”. This initiative provides "article-level metrics" on every article in their database to help users determine the value of that article to them and to the scientific community in general. Importantly, they provide additional and regularly updated context to the article, which currently includes data on citations, online usage, social bookmarks, comments, notes, blog posts about the article, and ratings of the article. Social network analysis The technique of Social Network Analysis (SNA) is able to assist in describing and understanding the patterns of participant interaction in Networked Learning environments . SNA is a research methodology that seeks to identify underlying patterns of social relations based on the way actors are connected with each other. De Laat et al (2007) propose that interactions among participants in Networked Learning communities may be relatively easily mapped out and explored using SNA, and provide additional useful analytical data about the activity and relationships of members in online learning communities . SNA may help in asking further questions about the nature of learning in online communities, because it provides a new way of viewing participants’ activities. It may also help to confirm or contextualize conclusions and interpretations about participants’ behavior in these environments gathered using existing analytical techniques. Using SNA, the social environment of participants in the Peer 2 Peer University can be mapped as well as the patterns of relationships among interacting members. This information can be used to validate the initial conceptual mapping done based on literature research. Especially interesting will be the intensity of the contact between different members, and the roles individual members fulfill. Moreover, an initial analysis focused on user participation may be used at the end of the research to determine the influence of the created artifact on motivation. Interviews Through interviews with selected semantic web developers and scientists (first cycle), and potential employers and computer scientists (second cycle), will we be able to evaluate feasibility and quality of technical solutions, and get insight into trust parameters that need to be addressed in the solution. The strength of an interview is that it describes real world situations . The weakness of this instrument lies in the possible bias of respondents. Interviews will be conducted in a semi-structured format. Interviews will be tape-recorded, and transcripts will be made.
Transcripts will be sent to the interviewees for approval. Interviewees will be selected based on perceived experience in the field of educational game design, and their history of relevant publications. Expert survey A first instrument that will be used for an expert survey, is the questionnaire with closed questions that make use of a Likert scale. A larger group of experts will be questioned; colleagues, national and international experts, from the corporate, governmental and academic sectors. Results will be summarized and analyzed through statistical analysis. A second instrument that will be used for an expert survey, is the Delphi Method; the systematic, interactive forecasting method which relies on a panel of independent experts . A threeround Delphi study will be employed to evaluate the semantic model, algorithm, and societal implications. A report will be written about the results. Literature review Several literature reviews will be conducted; one with a focus on peer-based learning in online communities, one on semantic annotation of concepts relating with reputation, quality, and user interest, and one on existing trust and reputation systems. For categorizing the reviewed literature, a framework will be used to define the (a) goal, (b) coverage, (c) organization, and (d) audience for the study. The results are organized conceptually and the relevant works are presented, in accordance to the framework used by Cooper . Literature will be drawn from journals, conference papers and books (dissertations, research reports, etcetera).
3.6 Scope of the research
This research will come up with a novel approach to reputation profiling to be used in peerbased online communities. The developed semantic model and system will not be a finished product, and most likely will never be. Rather, if the potential of the research is clear to a sufficiently large number of practitioners and researchers, then further research and development is likely to happen. This research will mainly focus on aspects not dealt (satisfactorily) in this research. Questions that are addressed in the research and will be not extensively researched include: If a person has different reputations on different, but similar communities, how can this person aggregate and merge these profiles? If someone joins a community, how can the existing reputation information, acquired in another community, be integrated? How does ‘knowledge’ or expertise degrade over time? How should reputation profiles be dealt with, who will manage the information, and how will this happen? What are the best interfaces to improve usage of the system?
3.7 Transferability, acceptance and dissemination of research
In order to ensure continued development on reputation in peer-based learning environment, several measures will be taken that will improve the transferability, acceptance, and dissemination.
• • • •
Open source. This research is inspired by open source economics and organization. Therefore, it would be contrary to belief when research results or created artifacts are not shared using no or an open source license. Learning community. If the idea catches on, the learning community that emerges will both develop and use the reputation system. Open standards. Following from the above, open standards, and especially standards for semantic annotation and concept mapping, are used to make the semantic model. Master students. The intention is to involve Master students working on their thesis in the all stages of the research, especially when conducting the case study analyses. VOCamps and conferences will be visited and research papers published to address the Semantic Web community as well as the educational technology community.
The research, as explained above, contains two cycles, aimed at developing a semantic model (first cycle), and the reputation algorithm (second cycle). Even though the cycles may seem rather distinct, new insights can affect both artifacts, and therefore the cycles cannot be seen separately. The planning below uses the steps and results defined in these cycles. The outline below contains scheduled research activities and their expected results for the four-year project.
Time jul-2009 aug-2009 sep-2009 okt-2009 nov-2009 dec-2009 jan-2010 feb-2010 mrt-2010 apr-2010 mei-2010 jun-2010 jul-2010 aug-2010 sep-2010 Activity General literature review Expected results Literature review on web-based learning, selfdirected learning
Draft research proposal
VOCamp Paris Finished research proposal Literature review on p2p learning
Development of conceptual model for User Interest First research proposal, as well as first chapter of dissertation First draft of conceptual model focused on motivation and typical activities Second draft of conceptual model adding relevance to the typical activities Overview of typical interactions, and intensity of interactions, without any reputation system A boy or a girl
Social network analysis on P2PU.org
nov-2010 dec-2010 jan-2011
Literature review on semantics for p2p learning Case study P2PU.org (interviews)
Overview of models and existing ontologies for expertise, trust, and more to learn from/reuse This case study will involve interviews with the main developers of the platform, and will evaluate the usefulness of individual elements and relations described in the model. Article Level Metrics may be used to improve definitions and relations defined in the model Design artifact:Development of the semantic model for reputation in p2p learning environments An overview of existing trust and reputation algorithms, with their merits and deficits. The existing systems will be matched with the requirements described in the semantic model to determine the value of various approaches and develop a new approach. This will shed light on the requirements on the reputation system regarding trust in such a system. Provides insight in user requirements concerned with privacy and personal information as well as online representation and ownership of information. Artifact: initial algorithm Artifact: theory on online reputation in the knowledge economy Artifact: improved semantic model Artifact: reputation system (that includes the algorithm and the semantic model) Improved reputation system (testing algorithm and ability to game the system) Improved reputation system (applicability in another context) Improved reputation system (applicability in another context) Improved reputation system (scalability of the system) Improved reputation system (external validity and trust in the system) Improved reputation system (internal validity and trust by reputed agents) Overview of typical interactions, and intensity of interactions, with the reputation system
Case study PLoS One Analyzing results
apr-2011 mei-2011 Literature review on online trust and reputation systems
Interviews with HRM managers on jobs.stackoverflow.com Interviews with members of P2PU.org
okt-2011 nov-2011 dec-2011 jan-2012 feb-2012 mrt-2012 apr-2012 mei-2012 jun-2012 jul-2012
Development of reputation system
Deployment of the system in P2PU.org Deployment of the system in StackOverflow Deployment of the system in PLoS One Conducting Delphi study Interviews with HRM managers on jobs.stackoverflow.com Interviews with members of P2PU.org Social network analysis on P2PU.org
aug-2012 sep-2012 okt-2012 nov-2012 dec-2012 jan-2013 feb-2013 mrt-2013 apr-2013 mei-2013 jun-2013 jul-2013 Aggregating results and writing dissertation Initial draft of dissertation
Final, revised document
List of references
           S. Downes, "Places to Go : Connectivism & Connective Knowledge," Innovate, vol. 5, 2008, p. 6. J. Bransford, A. Brown, and R. Cocking, How people learn: Brain, mind, experience, and school, Washington DC: National Academies Press, 1999. J. Brown and R. Adler, "Minds on fire: Open education, the long tail, and learning 2.0," Educause Review, vol. 43, 2008, p. 16. G. Siemens, "Connectivism - A learning theory for the digital age," International Journal of Instructional Technology and Distance Learning, vol. 2, 2005, p. 3–10. D. Wiley and E. Edwards, "Online self-organizing social systems: The decentralized future of online learning," Quarterly Review of Distance Education, vol. 3, 2002, p. 33. A. Koohang and K. Harman, "Advancing Sustainability of Open Educational Resources," Issues in Informing Science and Information Technology, vol. 4, 2007. S. Downes, "Models for sustainable open educational resources," Interdisciplinary Journal of Knowledge and Learning Objects, vol. 3, 2007, p. 29–44. D. Wiley, "On the sustainability of open educational resource initiatives in higher education," vol. 26, 2006, p. 21. I. Tuomi, "Open Educational Resources: What they are and why do they matter," October, available at: www. oecd. org/edu/oer, 2006, p. 44. J. Hylén, "Open educational resources: Opportunities and challenges," Open Education, 2006, p. 11. R.G. Baraniuk, "Challenges and Opportunities for the Open Education Movement: A Connexions Case Study," Opening up education - The collective advancement of education through open technology, open content, and open knowledge, T. Iiyoshi and M.V. Kumar, Cambridge, MA: MIT Press, 2008, pp. 229-247. J. Hylen, Giving Knowledge for Free: The Emergence of Open Educational Resources, Paris: 2007. S. Gurell, Open Educational Resources Handbook for Educators, 2008. M. Mcalpine, "E-portfolios and Digital Identity: some issues for discussion," ELearning, vol. 2, 2005, p. 378. P. van Der Zanden, The Facilitating University, Delft: Eburon Uitgeverij BV, 2009. J. Schmidt, C. Geith, S. Håklev, and J. Thierstein, "Peer-To-Peer Recognition of Learning in Open Education," The International Review of Research in Open and Distance Learning, vol. 10, 2009. S. Downes, The future of online learning, 2008. C. Davidson and D. Goldberg, "The future of learning institutions in a digital age," The MIT Press, 2009, p. 81. J.P. Schmidt and M. Surman, Open sourcing education: Learning and wisdom from the iSummit 2007, 2007. S. Downes, "The future of online learning," Online Journal of Distance Learning Administration, vol. 1, 1998. S. Downes, "E-learning 2.0," eLearn, vol. 2005, 2005, p. 1. J. Kenway, E. Bullen, and S. Robb, "The Knowledge Economy, the Techno-preneur and the Problematic Future of the University," Policy Futures in Education, vol. 2, 2004, p. 330. W. Snyder and E. Wenger, "Communities of Practice," Lessons learned from Auburn Hills. …, 1999. 61
           
                    
E. Wenger, R. McDermott, and W. Snyder, Cultivating communities of practice: A guide to managing knowledge, Boston, MA: HBS Press, 2002. A. Ardichvili, "Learning and Knowledge Sharing in Virtual Communities of Practice: Motivators, Barriers, and Enablers," Advances in Developing Human Resources, vol. 10, 2008, pp. 541-554. M. Soekijad, M.a. Huis in 't Veld, and B. Enserink, "Learning and knowledge processes in inter-organizational communities of practice," Knowledge and Process Management, vol. 11, 2004, pp. 3-12. C. Johnson, "A survey of current research on online communities of practice," The Internet and Higher Education, vol. 4, 2001, pp. 45-60. J. Storck, "Communities of practice and organizational performance," International Business, 2001, pp. 831-841. E. Wenger, "Communities of practice and social learning systems," Organization, vol. 7, 2000, pp. 225-246. B. Allan and D. Lewis, "The impact of membership of a virtual learning community on individual learning careers and professional identity," British Journal of Educational Technology, vol. 37, 2006, pp. 841-852. M.J. Hannafin, K.M. Hannafin, S.M. Land, and K. Oliver, "Grounded practice and the design of constructivist learning environments," Educational Technology Research and Development, vol. 45, 1997, pp. 101-117. G. Lefoe, "Creating constructivist learning environments on the web: the challenge in higher education," Development, 1997, pp. 453-464. D.H. Jonassen, "Objectivism versus constructivism: Do we need a new philosophical paradigm?," Educational Technology Research and Development, vol. 39, 1991, pp. 514. J.R. Savery and T.M. Duffy, "Problem Based Learning: An instructional model and its constructivist framework," Constructivist learning environments: case studies in instructional design, B.G. Wilson, 2001, pp. 135-151. S.A. Barab and T. Duffy, "From Practice Fields to Communities of Practice," Learning, 2000. M. Zarb, "Modelling participation in virtual communities-of-practice," 2006. A.E. Veldhuis-Diermanse, H.J. Biemans, M. Mulder, and H. Mahdizadeh, "Analysing Learning Processes and Quality of Knowledge Construction in Networked Learning," The Journal of Agricultural Education and Extension, vol. 12, 2006, pp. 41-57. M.D. Laat, "Networked Learning," 2006, pp. 7-9. M.D. Laat and V.I. Lally, "Complexity, theory and praxis: Researching collaborative learning and tutoring processes in a networked learning community," 2003, pp. 7-39. P.D. Vries, "Networked Learning for Lifelong Learners," Journal of documentation, 2008, p. 26. I. Tuomi, "Learning in the Age of Networked Intelligence," European Journal of Education, vol. 42, 2007, pp. 235-254. B. Zimmerman, "Attaining self-regulation: A social cognitive perspective," Handbook of self-regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, 2000, pp. 13-40. D. Schunk and B. Zimmerman, "Self-regulation and learning," Handbook of Psychology, Volume 7, Educational Psychology, W.M. Reynolds, G.E. Miller, and I.B. Weiner, Wiley, 2003, pp. 59-78. W. Veen, J.V. Staalduinen, and T.A. Hennis, "Informal self-regulated learning in corporate organizations (submitted)," Fostering Self-regulated learning through ICTs (forthcoming), G. Dettori and D. Persico, Genova, Italy: Institute for Educational Technologies Italy's National Research Council, 2010.
   
              
B. Zimmerman, "Becoming a self-regulated learner: An overview," Theory into practice, 2002. J. Randi, "Teachers as self-regulated learners," The Teachers College Record, 2004. J. Meece, "The role of motivation in self-regulated learning," Self-regulation of learning and performance: Issues …, 1994. S. Paris and P. Winograd, "The role of self-regulated learning in contextual teaching: Principles and practices for teacher preparation," Preparing Teachers to Use Contextual Teaching and Learning Strategies to Improve Student Success in and beyond School, 2003. L. Kremer-Hayon and H. Tillema, "Self-regulated learning in the context of teacher education," Teaching and Teacher Education, 1999. G. Hardin, "The Tragedy of the Commons," Journal of Natural Resources Policy Research, vol. 1, 2009, pp. 243-253. E. Raymond, "The cathedral and the bazaar," Knowledge, Technology & Policy, vol. 12, 1999, pp. 23-49. P. Himanen, The hacker ethic and the spirit of the information age, Random House Inc. New York, NY, USA, 2001. E.V. Hippel and G.V. Krogh, "Open Source Software and the "Private-Collective" Innovation Model: Issues for Organization Science," Organization Science, vol. 14, 2003, pp. 209-223. Y. Benkler, The wealth of networks, New Haven, CT: Yale University Press, 2006. M. Mauss, The gift; Forms and Functions of Exchange in Archaic Societies, New York: The Norton Library, 1967. M. McLure-Wasko and S. Faraj, "Why Should I Share? Examining Social Capital and Knowledge Contribution in Electronic Networks of Practice," MIS Quarterly, vol. 29, 2005. P. Kollock, "The economies of online cooperation: Gifts and public goods in cyberspace," Communities in cyberspace, M.A. Smith and P. Kollock, New York: Routledge, 1997, pp. 220-239. T.A. Hennis and G.L. Kolfschoten, "Understanding Social Mechanisms in Online Communities (submitted)," Group Decision and Negotiation 2010, Delft: 2010. J. Lalley and R. Miller, "The Learning Pyramid: Does It Point Teachers in the Right Direction?.," Education, vol. 128, 2007, p. 16. J.S. Brown, "The Social Life of Learning: How can Continuing Education be Reconfigured in the Future?," Continuing Higher Education Review, vol. 66, 2002, pp. 50-69. Y. Wang and J. Vassileva, "Trust and reputation model in peer-to-peer networks," Proceedings of the 3rd International Conference …, 2003. T.L. Wentling, C. Waight, J. Gallaher, J.L. Fleur, C. Wang, A. Kanfer, and L.S. Group, "e-learning - A Review of Literature," 2000, pp. 1-73. L. Vu, T.G. Papaioannou, and K. Aberer, "Synergies of Different Reputation Systems: Challenges and Opportunities," 2009 World Congress on Privacy, Security, Trust and the Management of e-Business, Saint John, Canada: IEEE Computer Society, 2009, pp. 218-226. M. McLure-Wasko and S. Faraj, "Why should I share? Examining social capital and knowledge contribution in electronic networks of practice," MIS Quarterly, vol. 29, 2005, pp. 35-57. P. Kollock, "The Economies of Online Cooperation: Gifts and Public Goods in Cyberspace," Communities in cyberspace, M. Smith and P. Kollock, 1999.
    
             
J. Lampel and A. Bhalla, "The Role of Status Seeking in Online Communities: Giving the Gift of Experience," Journal of Computer-Mediated Communication, vol. 12, 2007, p. article 5. A. Berlanga, E. Rusman, M. Bitter-Rijpkema, and P. Sloep, "Guidelines to Foster Interaction in Online Communities," Learning Network Services for Professional Development, R. Koper, Berlin, Heidelberg: Springer, 2009, pp. 27-42. S. Downes, "The future of online learning - 10 years later," Half an Hour (Blog), vol. 1, 2008. I. Tuomi, "Learning in the Age of Networked Intelligence," European Journal of Education, vol. 42, 2007, pp. 235-254. J. Preece and D. Maloney-Krichmar, "Online communities: focusing on sociability and usability," Handbook of human-computer interaction, J. Jacko and A. Sears, Mahwah, New Jersey: Lawrence Erlbaum Associates Inc. Publishers, 2003, p. 596– 620. B. Matthews, "Semantic Web Technologies," 2005, p. 19. J. Hendler, "The Dark Side of the Semantic Web," IEEE Intelligent Systems, vol. 22, 2007, pp. 2-4. D. Siegel, Pull: The Power of the Semantic Web to Transform Your Business, Portfolio Hardcover, 2009. E. Chang, F. Hussain, and T. Dillon, "Reputation Ontology for Reputation Systems," Lecture notes in computer science, 2005. J. Golbeck and J. Hendler, "Reputation network analysis for email filtering," Proceedings of the First Conference on Email and …, 2004. J. Golbeck and J. Hendler, "Accuracy of Metrics for Inferring Trust and Reputation in Semantic Web-based Social Networks," Lecture Notes in Computer Science, 2004. L. Vercouter, S. Casare, and J. Sichman, "An experience on reputation models interoperability based on a functional ontology," Proceedings of The IJCAI-07, 2007, pp. 617-622. S. Casare and J. Sichman, "Towards a functional ontology of reputation," Proceedings of the fourth international joint …, 2005. P.J. Windley, K. Tew, and D. Daley, "A Framework for Building Reputation Systems," WWW2007, 2007. J. Golbeck, C. Park, and J. Hendler, "Inferring Reputation on the Semantic Web," A Journal On The Theory Of Ordered Sets And Its Applications, 2004. M. Grobelnik and D. Mladenic, "Ontology Generation from Social Networks," Social Networks, 2009. B. Aleman-meza, U. Bojars, H. Boley, J.G. Breslin, M. Mochol, A. Polleres, and A.V. Zhdanova, "Combining RDF Vocabularies for Expert Finding," Science, 2005. United Nations General Assembly, "Universal Declaration of Human Rights," Wikipedia, 1948. G. Stahl, "Meaning and interpretation in collaboration," Designing for Change in Networked Learning Environments: Proceedings of the International Conference on Computer Support for Collaborative Learning (CSCL ‘03), B. Wasson, S. Ludvigsen, and U. Hoppe, Bergen: Kluwer Academic Publishers, 2003, pp. 523-532. T. Caswell, S. Henson, M. Jensen, and D. Wiley, "Open Educational Resources : Enabling universal education," International Review of Research in Open and Distance Learning, vol. 9, 2008, pp. 1-11. Opening up education - The collective advancement of education through open technology, open content, and open knowledge, Cambridge, MA: MIT Press, 2008.
                    
M. Pascual, G. Geser, A. Gruber, S.R. Forschungsgesellschaft, P. Baumgartner, V. Naust-, L. Pullich, S. Lamminnen, A. Koivisto, and M. Group, "The concept of open educational resources as instrument for implementing lifelong learning strategies at higher and further education institutions," Source, 2006, pp. 1-8. U. Klotz, "The challenges of the New Economy *," 1999, pp. 1-24. B. Skinner, "Are theories of learning necessary?.," Psychological review, vol. 57, 1950, p. 193–216. E. Thorndike, An introduction to the theory of mental and social measurements, Teacher's College, Columbia University, 1913. D.a. Norman, "Twelve Issues for Cognitive Science," Cognitive Science, vol. 4, 1980, pp. 1-32. A. Newell and H.A. Simon, Human problem solving, Englewood Cliffs, NJ: PrenticeHall, 1972. J.R. Anderson, "Acquisition cognitive skills," Psychological review, vol. 89, 1982, pp. 369-406. A. de Groot, "Thought and Choice in Chess (rev. translation)," Mouton and Company, The Hague, 1965. K. Ericsson and N. Charness, "Expert performance: Its structure and acquisition.," American psychologist, vol. 49, 1994, p. 725–747. B. Cole, "Characterizing On-line Communication: A First Step.," 1996. J. Lave and E. Wenger, Situated learning: Legitimate peripheral participation, Cambridge Univ Pr, 1991. J. Lave, Cognition in practice: Mind, mathematics, and culture in everyday life, Cambridge Univ Pr, 1988. B. Rogoff, "Apprenticeship in thinking: Cognitive development in social context," 1990. J. Tudge and B. Rogoff, "Peer influences on cognitive development: Piagetian and Vygotskian perspectives," Lev Vygotsky: Critical assessments, 1999. L. Vygotsky and M. Cole, Mind in society: The development of higher psychological processes, 1978. J. Bruner, "The narrative construction of reality," Critical inquiry, vol. 18, 1991, p. 1– 21. J. Piaget and M. Cook, The origins of intelligence in children, International Universities Press New York, 1952. A. Amory and R. Seagram, "Educational game models: conceptualization and evaluation," Journal of Higher Education, vol. 17, p. 206–217. B.V. Hout-Wolters, R. Simons, and S. Volet, "Active Learning: Self-directed learning and independent work," Springer, R. Simons, Kluwer Academic Publishers, 2000, pp. 21-36. L. Rieber, "Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games," Educational technology research and development, vol. 44, 1996, p. 43–58. W. Bouman, T. Hoogenboom, R. Jansen, M. Schoondorp, B. de Bruin, and A. Huizing, The realm of sociality: notes on the design of social software, Amsterdam: 2007. M. Ackerman, J.M. Carroll, G. Demichelis, M. Huysman, B. Wellman, and V. Wulf, "Communities and Technologies: An Approach to Foster Social Capital?," Practice, 2004, pp. 406-408.
 H. Allert, "Coherent Social Systems for Learning: An Approach for Contextualized and Community Centred Metadata," Journal of Interactive Media in Education, vol. 2004, 2004, pp. 1-30.  E. Dale and B. Nyland, "Cone of learning," Educational Media, 1960.  S. Paavola, L. Lipponen, and K. Hakkarainen, "Models of Innovative Knowledge Communities and Three Metaphors of Learning," Review of Educational Research, vol. 74, 2004, pp. 557-576.  G. Stahl, T. Koschmann, and D. Suthers, "Computer-supported collaborative learning: An historical perspective," Cambridge handbook of the learning sciences, S. Sawyer, Cambridge, UK: Cambridge University Press, 1999.  W. Veen, H. Lukosch, and P. de Vries, "Improving organizational learning through Networked Learning," GI2008, 2008, pp. 22-31.  C. Jones and C. Steeples, "Perspectives and issues in networked learning," Networked learning, Springer-Verlag New York, Inc., 2001, p. 14.  G. Siemens, Knowing knowledge, Lulu. com, 2006.  K. Illeris, "Towards a contemporary and comprehensive theory of learning," International Journal of Lifelong Education, vol. 22, 2003, p. 396–406.  Y. Benkler, "Common Wisdom: Peer Production of Educational Materials," 2005.  T.A. Hennis, E. Sjoer, and W. Veen, "The future of open courseware; A Case Study," Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications, 2008, pp. 686-696.  G.H. Brundtland, "Our Commons Future," 1987.  K. Hargroves and M. Smith, "The Natural Advantage of Nations," Business Opportunities, Innovation and Governance in the 21st Century, London: Earthscan/James&James, 2005, pp. 7-25.  L. Ripamonti, F. De Cindio, and M. Benassi, "Online communities sustainability: some economic issues," The Journal of Community Informatics, vol. 1, 2005, p. 63–78.  J. Giles, "Internet encyclopaedias go head to head.," Nature, vol. 438, 2005, pp. 9001.  M.A. Wirth, "An analysis of international quality management approaches in elearning: Different paths, similar pursuits," Handbook on Quality and Standardisation in E-Learning, U. Ehlers and J.M. Pawlowski, Berlin, Heidelberg: Springer, 2006, pp. 97-108.  J. Milam, R.A. Voorhees, and A. Bedard-Voorhees, "Assessment of online education: Policies, practices, and recommendations," New Directions for Community Colleges, 2004, pp. 73-85.  R. Voorhees, Measuring What Matters: Competency-Based Learning Models in Higher Education: new directions for institutional research, Lavoisier, 2001.  S. Downes, Open Source Assessment.  J. Koh, Y. Kim, B. Butler, and G. Bock, "Encouraging participation in virtual communities," Communications of the ACM, vol. 50, 2007, pp. 68-73.  B.S. Butler, When is a Group Not a Group: An Empirical Examination of Metaphors for Online Social Structure, Pittsburgh: 1999.  T.A. Hennis, "Platform voor promovendi - Valkuilen op de werkplaats," Intellektueel Kapitaal, 2009, pp. 40-42.  T.A. Hennis, "Platform voor promovendi - Lessons learned leidt tot achtstappenplan," Intellektueel Kapitaal, 2009, pp. 36-40.  F. Ortega, J.M. Gonzalez-Barahona, and G. Robles, "On the inequality of contributions to Wikipedia," Proceedings of the 41st Annual Hawaii International Conference on System Sciences, 2008.
 J. Preece and D. Maloney-Krichmar, "Online communities: Design, theory, and practice," Journal of Computer-Mediated Communication, vol. 10, 2005.  M.J. Culnan and B. College, "Online Communities: Infrastructure, Relational Cohesion, and Sustainability," Computerizations Movements and Technology Diffusion: From Mainframes to Ubiquitous Computing, M.S. Elliot and K.L. Kraemer, New Jersey: Information Today, 2008, pp. 239-259.  T. Yu, L. Lu, and T. Liu, "Exploring factors that influence knowledge sharing behavior via weblogs," Computers in Human Behavior, vol. 26, 2010, pp. 32-41.  T.D. Moore and M.A. Serva, "Understanding Member Motivation for Contributing to Different Types of Virtual Communities: A Proposed Framework," SIGMIS-CPR '07: Proceedings of the 2007 ACM SIGMIS CPR conference on 2007 computer personnel doctoral consortium and research conference, New York: ACM, 2007, pp. 153-158.  P. Hendriks and D. Vriens, "Knowledge-based systems and knowledge management: friends or foes?," Information & Management, vol. 35, 1999, pp. 113-125.  H. Lee and B. Choi, "Knowledge management enablers, processes, and organizational performance: an Integrative View and Empirical Examination," Journal of Management Information Systems, vol. 20, 2003, pp. 179-220.  H. Lee, "The effects of socio-technical enablers on knowledge sharing: an exploratory examination," Journal of Information Science, vol. 34, 2008, pp. 742-754.  H.L. Rheingold, "Virtual Communities and the WELL," GNN Magazine, 1993.  E. Pearson, "Digital gifts: Participation and gift exchange in Livejournal communities," First Monday, vol. 12, 2007.  M. McLure-Wasko, R. Teigland, and S. Faraj, "The provision of online public goods: Examining social structure in an electronic network of practice," Decision Support Systems, vol. 47, 2009, pp. 254-265.  M. Endres, S. Endres, S. Chowdhury, and I. Alam, "Tacit knowledge sharing, selfefficacy theory, and application to the Open Source Community," Journal of Knowledge Management, vol. 11, 2007, pp. 92-103.  E.V. Hippel, Democratizing Innovation, Cambridge, MA: MIT Press, 2005.  C. Lattemann and S. Stieglitz, "Framework for Governance in Open Source Communities," Proceedings of the 38th Annual Hawaii International Conference on System Sciences, vol. 00, 2005, pp. 192a-192a.  F. Labalme and K. Burton, "Reputation capital and exchange mechanisms," 2002.  Y. Wang, "An overview of online trust: Concepts, elements, and implications," Computers in Human Behavior, vol. 21, 2005, pp. 105-125.  M. Kinateder and K. Rothermel, "Architecture and algorithms for a distributed reputation system," Lecture notes in computer science, 2003, p. 1–16.  D. Gambetta, "Can We Trust Trust?," Trust: Making and Breaking Cooperative Relations, D. Gambetta, 2000, pp. 213-237.  L. Mui, M. Mohtashemi, and A. Halberstadt, "Notions of reputation in multi-agents systems," Proceedings of the first international joint conference on Autonomous agents and multiagent systems part 1 - AAMAS '02, 2002, p. 280.  J. Sabater and C. Sierra, "Review on Computational Trust and Reputation Models," Artificial Intelligence Review, vol. 24, 2005, pp. 33-60.  P. Resnick, R. Zeckhauser, E. Friedman, and K. Kuwabara, "Reputation Systems," Communications of the ACM, vol. 43, 2000, pp. 45-48.  F. Squazzoni, "Review of Reputation in Artificial Societies: Social Beliefs for Social Order," 2004.
 a. Abdul-Rahman and S. Hailes, "Supporting trust in virtual communities," Proceedings of the 33rd Annual Hawaii International Conference on System Sciences, vol. 00, 2000, p. 9.  Y. Tan and W. Thoen, "Towards a generic model of trust in electronic commerce," International Journal of Electronic Commerce, vol. 5, 2000, pp. 61-74.  A. Josang, "Trust and Reputation Systems," FOSAD 2006/2007, R. Aldini and R. Gorrieri, Heidelberg: Springer-Verlag, 2007, pp. 209-245.  S.D. Kamvar, M.T. Schlosser, and H. Garcia-Molina, "The Eigentrust algorithm for reputation management in P2P networks," Proceedings of the twelfth international conference on World Wide Web - WWW '03, 2003, p. 640.  N. Gal-Oz, T. Grinshpoun, E. Gudes, and A. Meisels, "CROSS-COMMUNITY REPUTATION: POLICIES AND ALTERNATIVES," IADIS Int. Conf. on Web Based Communities (WBC2008), 2008, pp. 197-201.  F.R. Farmer and B. Glass, Building Web Reputation Systems, O'Reilly Media, 2010.  S. Brin and L. Page, "The anatomy of a large-scale hypertextual Web search engine," Computer Networks and ISDN Systems, vol. 30, 1998, pp. 107-117.  D. Lewandowski, "Web Searching: A Quality Measurement Perspective," 2007.  P. Resnick, R. Zeckhauser, J. Swanson, and K. Lockwood, "THE VALUE OF REPUTATION ON EBAY: A CONTROLLED EXPERIMENT," Experimental Economics, vol. 9, 2006, p. 79–101.  T. Malone and R. Laubacher, "The E-lance economy," Harvard Business Review, vol. 76, 1998, pp. 145-152.  D. Florzak, "Are you ready for the e-lance economy?," TECHNICAL COMMUNICATION-WASHINGTON-, vol. 49, 2002, p. 162–170.  S. Saha, S. Saint, and D. Christakis, "Impact factor: a valid measure of journal quality?," JOURNAL-MEDICAL LIBRARY …, 2003.  F. Hecht, B. Hecht, and A. Sandberg, "The Journal “Impact Factor” A Misnamed, Misleading, Misused Measure," Cancer Genetics and Cytogenetics, 1998.  M. Bordons, M. Fernández, and I. Gómez, "Advantages and limitations in the use of impact factor measures for the assessment of research performance," Scientometrics, vol. 53, 2002, pp. 195-206.  J. Bollen, M.A. Rodriguez, and H. Van de Sompel, "Journal status," Scientometrics, vol. 69, 2006, pp. 669-687.  A. Montenegro-Montero, On PLoS' article-level metrics, 2009.  A. Gami, V. Montori, and N. Wilczynski, "Author self-citation in the diabetes literature," Canadian Medical …, 2004.  D. Harel and B. Rumpe, "Meaningful Modeling : What's the Semantics of "Semantics"?," Computer, 2004, pp. 64-72.  C. Neylon and S. Wu, "Article-level metrics and the evolution of scientific impact.," PLoS biology, vol. 7, 2009, p. e1000242.  N. Shadbolt, T. Berners-Lee, and W. Hall, "The Semantic Web Revisited," IEEE Intelligent Systems, vol. 21, 2006, pp. 96-101.  J. Breslin, A. Passant, and S. Decker, The Social Semantic Web, Heidelberg: SpringerVerlag, 2009.  T. Berners-Lee, J. Hendler, and O. Lassila, "The Semantic Web," Scientific American, vol. 284, 2001, pp. 34-43.  L. Li and I. Horrocks, "A software framework for matchmaking based on semantic web technology," Proceedings of the twelfth international conference on World Wide Web - WWW '03, 2003, p. 331.  C. Shirky, "The Semantic Web, Syllogism, and Worldview."
 J. Hendler, "Semantics and the Network Effect: A little semantics goes a long way," IEEE Intelligent Systems, 2002, pp. 2002-2002.  J. Hendler, "It's just a matter of semantics," Fortune Tech, 2009.  R.D. Neufville, "Thesis definition and preparation," 2002, p. 53.  J. Anderson and L. Rainie, "The Impact of the Internet on Institutions in the Future," 2010, p. 35.  E. Cuba and Y. Lincoln, "Competing paradigms in qualitative research," Handbook of qualitative research, N. Denzin and Y. Lincoln, Thousand Oaks, CA: Sage Publications, Inc, 1994, pp. 105-117.  W. Orlikowski and J. Baroudi, "Studying information technology in organizations: Research approaches and assumptions," Informations Systems Research, vol. 2, 1991, pp. 1-28.  A.R. Hevner, S.T. March, J. Park, and S. Ram, "Design Science in Information Systems Research," MIS Quarterly, vol. 28, 2004, pp. 75-105.  R. Flood, Liberating Systems Theory, New York: Plenum Press, 1990.  R. Flood, "Liberating Systems Theory: Towards Critical Systems Thinking," Human Relations, vol. 43, 1990, pp. 49-75.  H. Sol, "Simulation in Information Systems Development," 1982.  I.I. Mitroff, F. Betz, L.R. Pondy, and F. Sagasti, "On Managing Science in the Systems Age: Two Schemas for the Study of Science as a Whole Systems Phenomenon," Interfaces, vol. 4, 1974, pp. 46-58.  S.T. March and G.F. Smith, "Design and natural science research on information technology," Decision Support Systems, vol. 15, 1995, pp. 251-266.  B. Kuechler and V. Vaishnavi, "On theory development in design science research: anatomy of a research project," European Journal of Information Systems, vol. 17, 2008, pp. 489-504.  H.K. Klein and M.D. Myers, "A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems h," MIS Quarterly, vol. 23, 1999, pp. 67-93.  C. Chorus, B.V. Wee, and S. Zwart, "TPM-catalogue of Concepts, Theories and Methods," 2010, p. 278.  J. Venable, "A framework for Design Science research activities," 2006 Information Resources Management Association International Conference, M. Khosrow-Pour, Washington DC: Idea Group Publishing, 2006, pp. 184-187.  R. Yin, Case study research: Design and methods, Sage Pubns, 2008.  K. Lewin, Resolving social conflicts: and, Field theory in social science, American Psychological Association Washington, DC, 1997.  M. Laat, V. Lally, L. Lipponen, and R. Simons, "Investigating patterns of interaction in networked learning and computer-supported collaborative learning: A role for Social Network Analysis," International Journal of Computer-Supported Collaborative Learning, vol. 2, 2007, pp. 87-103.  J. Gubrium and J. Holstein, Handbook of interview research: Context \& method, Sage Publications, Inc, 2002.  H. Linstone, M. Turoff, and others, The Delphi method: Techniques and applications, Addison-Wesley Publ., 1975.  H. Cooper, "Organizing knowledge syntheses: A taxonomy of literature reviews," Knowledge, Technology & Policy, vol. 1, 1988, p. 104–126.