Project definition 

Sustaining peer‐based learning through reputation modeling 

Version - June 2nd , 2010

Thieme Hennis

Supervisors: Wim Veen & Stephan Lukosch

Index
Index  2 
1  Introduction  4 
1.1  Developments in open education  4 
1.2  Gift culture, reciprocity, and reputation  6 
1.3  Semantics of reputation  7 
1.4  Problem statement  8 
1.5  Outline of the proposal  9 
2  Literature review  10 
2.1  Learning  11 
2.1.1  Learning theories and pedagogies  11 
2.1.2  Learning scenarios  16 
2.1.3  Conclusions  17 
2.2  Sustainability  18 
2.2.1  Sustainability of Open Educational Resources  18 
2.2.2  Sustainability of online communities  21 
2.2.3  Conclusions on sustainability  22 
2.3  Reputation and trust  24 
2.3.1  Why reputation?  24 
2.3.2  Trust and reputation  25 
2.3.3  Context, transitivity, and cross‐community reputation  26 
2.3.4  Examples of trust and reputation systems  28 
2.3.5  Conclusions on trust and reputation  34 
2.4  Semantic web  38 
2.4.1  What is the Semantic Web?  38 
2.4.2  Semantic Web Technologies  39 
2.4.3  Relevance of the Semantic Web  40 
3  Scientific approach: philosophy, strategy, and planning  41 
3.1  Research questions  41 
3.1.1  Operational research questions  42 
3.2  Societal and scientific relevance  43 
3.3  Research philosophy and methodology  44 
3.3.1  Ontology and epistemology  46 
3.3.2  Methodology  46 
3.3.3  Summary of research philosophy  47 
3.4  Methodology: Design Science  48 
3.4.1  Guidelines  49 
3.4.2  Design Science activities  50 
3.5  Design cycles and instruments  51 
3.5.1  First cycle: developing the semantic model  51 
3.5.2  Second cycle: developing the reputation system  53 
3.5.3  Research instruments  55 
3.6  Scope of the research  57 
3.7  Transferability, acceptance and dissemination of research  57 

2
 3.8  Planning  58 
List of references  61 

3
1 Introduction
In 2008, George Siemens and Stephen Downes organized an online course on Connectivism
and Connective Knowledge[1]. Over 2200 persons worldwide actively participated in an
online, peer-based learning network. Feedback is an essential feature in learning[2]. Because
the organizers were unable to assess and give individual feedback to each of the students, they
motivated students to give peer-feedback, using technologies of all kind: virtual worlds,
blogging, commenting, RSS feed-readers, Moodle CMS, and much more.

Interestingly, this course was given for free, as part of a research project by the two
organizers. Since 2001, open education has been a linked with all kinds of initiatives that have
the objective of providing better access to learning by providing learning resources for free on
the Internet[3]. The course by Siemens and Downes, together with several other similar
initiatives the last couple of years, adds another element to the open education spectrum:
guidance and peer-support. Connectivist and networked learning approaches focus on the
ability of the network as a whole to learn, and to learn from the network[4].

CCK08 was an interesting initiative, but only partially shows how learning is evolving with
the Internet[5]. Even though it gave access to learning resources, and provided guidance, it
offered official accreditation to only a very limited group: only 25 (less than 2 percent of the
students) could be evaluated. Traditional methods for evaluation and assessment seem
inadequate for an online context.

1.1 Developments in open education 
The figure below shows the developments in Open Education.

Learning 
communities 
• P2PU.org 
Free online courses  • Peer‐based learning 
• Tools to support peer‐
• CCK08  assessment and open 
• Guidance by experts  accreditation? 
• Peer‐assessment 
• Limited accreditation 
• Tools to support social 
Open Educational  interaction and 
knowledge exchange 
Resources 
• Example: MIT OCW 
• Static and dynamic 
content 
• Tools to publish content 

Figure 1 - Developments in Open Education

Two important challenges are identified for the open education community:

4
• sustainability to the continuous ability to create, maintain, and share high-quality learning
resources[6,7,8,9,10,11,12], and
• assessment, accreditation, and recognition of learning achievements in a scalable and
sustainable way[4,5,13,14,15,16].
The proposed research intends to provide solutions to both of these challenges.

As the learning landscape is gradually transforming into more networked, online, and
interactive learning landscapes[1,3,5,10,17,18,19,20,21,22], we find ourselves facing
significant challenges on different levels. The roles of learning institutions is changing
rapidly[15,21,22], new skills are needed that emphasize on the learners’ capability to find
relevant information scattered in (online) networks [4,18,22].

Technology (most significantly, the Internet) offers many new opportunities for learning and
learning institutions, but important questions need to be answered:

“If we are going to imagine new learning institutions that are not based on the contiguity of
time and place—virtual institutions—we have to ask, what are those institutions and what
work do they perform? What does a virtual learning institution look like, who supports it,
what does it do? We know that informal learning happens, constantly and in many new ways,
because of the collaborative opportunities offered by social networking sites, wikis, blogs,
and many other interactive digital sources. But beneath these sites are networks and,
sometimes, organizations dedicated to their efficiency and sustainability. What is the
institutional basis for their persistence? If a virtual site spans many individuals and
institutions, who or what supports (in practical terms) the virtual site and by what
mechanisms?”[18]

We are currently moving toward a new future of open education and learning. Networked,
informal learning in online communities of practice is argued to become an important mode
of learning [23,24,25,26,27,28,29]. Participation in these online communities also have
positive influence on professional life:

“…through the process of engaging in a VLC, individuals may change their ‘horizons of
action’ leading to new learning and career trajectories. In particular, the study demonstrates
how membership of a VLC supported and enabled some individuals to transform their
learning careers and to make significant life changes. Other members developed their
learning careers in an incremental manner that led to increased innovation and professional
expertise.”[30]

If we turn our attention to the concerns mentioned by Davidson and Goldberg [18], we
identify a need for new models of organization of learning. Recent developments on learning
theories and pedagogies, including social constructivism[27,31,32,33],
Connectivism[4,16,21], problem-based learning[34], communities of
practice[23,24,28,29,35,36], active and networked learning[37,38,39,40,41], self-regulated
learning[42,43,44,45,46,47,48,49] describe learning as a intrinsically social activity that can
be supported by online communities.

5
1.2 Gift culture, reciprocity, and reputation 
Addressing the opportunities of learning in online communities, we must focus on the
question how these communities come to existence and are sustained. Inspiration may come
from the Open Source domain. In these self-organizing, online communities, responsibility to
locate, assemble, and contextualize learning resources is with the members themselves.
Although the tragedy of the commons [50] would suggest that such voluntary collaborations
are not sustainable over time, the emergence of the Internet, and specifically the Free/Libre
Open Source Software (FLOSS) movements, have shown peer-to-peer communications
technology’s ability to put people in symbiotic, “you answer my question, I’ll answer yours”
relationships. The gift culture described by ethnographers of the FLOSS movements such as
Raymond (1999), Himanen (2001), Benkler (2005), and von Hippel (2003) is one explanation
of this phenomenon[51,52,53,54]. Another reason for ‘free’ forms of knowledge exchange is
one of knowledge management: expertise exists across the community, so no individual
community member is overly burdened with the primary responsibility for answering
questions and providing feedback[5]. As problems arise related to the expertise of an
individual, that individual may or may not choose to provide help. If the community is of
sufficient size, the distribution of expertise and effort provides timely problem solving
support without unduly burdening any individual.

The inventor of the term ‘gift economy’, Marcell Mauss (1926), describes the gifts as a
particular form of bonding, of creating mutual dependencies:

The form usually taken is that of the gift generously offered; but the accompanying behavior
is formal pretence and social deception, while the transaction itself is based on obligation
and economic self-interest.[55]

Reciprocity, or the expectation to get something in return, is a central motivation in online

communities, where ‘gifts’ take the form of sharing expert knowledge, or the creation of
digital artifacts[25,56,57,58]. This important driver for knowledge exchange and participation
in online communities brings us to the question what kind of ‘return on investment’ a
community member expects when he or she shares knowledge.

Learning, in its most fundamental form, is aimed at survival. Darwin once famously said that
not the most intelligent, or strongest of species would survive, but the one most adaptable to
change. Even though adaptation is a form of intelligence, one could argue, the importance of
learning is made clear. It is a means to be able to cope with changes in the environment. For
most human beings, learning is aimed at getting in tune with society. Higher education is
aimed at preparing learners for professional work[2,59,60]. A rapidly changing society
therefore requires learners that can adapt quickly, and learn relevant things on demand, whilst
being able to de-learn irrelevant things[17,43]. Reciprocity, in the scenario of lifelong
learning in online communities, will take the form of getting relevant information on demand,
and developing a reputation that is useful in a professional context
[4,14,16,25,36,44,51,58,61,62,63].

6
Again, the Open Source community can be used as an example of how online reputation can
be used as an artifact to convey trust in a person, both for employers and between developers
[41,64,65,66,67,68,70]. Tuomi describes it as follows:

“For example, the social importance of formal educational certificates is now declining, as
the capabilities, interests, and reputations of people can be directly evaluated using
information and communication technologies. Instead of asking whether a job candidate has
a formal educational status, a potential employer can now review the candidate’s actual track
record, blog postings, and possibly e-portfolios. Already today people compile extensive life
histories using web logs and homepages. In some expertise areas, for example in computer
programming, employment opportunities often depend on a track record that can be
reconstructed by search engines and personal blogs. The digital identities of persons now
consist of their own representations of achievements and experiences, as well as reputations
that accumulate through the comments of others.”[41]

Reputation modeling in online communities could provide an online alternative to

institutional assessment methods and as an incentive for individuals to engage in all roles
needed to sustain peer-based learning environments. The question is then: “how can, and
should reputation be modeled in order to provide trustworthy representations of knowledge
and skills (as institutional accreditation aims to do), and provide incentive to share knowledge
and (partially) sustain peer-based online learning?”

There are two basic criteria for a reputation to be useful. First of all, the profiles that are
created in an online reputation system must be accurate. Secondly, the community as well as
the external world must recognize (or trust) the reputation. These requirements lead us to the
Semantic Web.

1.3 Semantics of reputation 
The Semantic Web is a large-scale effort to define terms, concepts, and relationships on the
Internet in machine-readable way, unambiguously[71,72,73]. Systems are being designed that
can use semantically annotated data in various ways, including semantic search,
matchmaking, automation, analysis, and more. The requirement of accuracy and external
validity of the proposed reputation system points in the direction of the Semantic Web as a
relevant domain to position the research (for development and publishing).

By definition, a reputation system that adheres to open semantic standards can more easily be
used in other contexts than a closed system. Acceptance and applicability are requirements for
an open reputation system, and following Semantic Web standards seem an obvious path to
follow. Some relevant existing semantic schemas describing trust and reputation have been
developed[63,74,75,76,77,78,79,80]. Despite the developments in this direction, none of the
schemas describes the interactions specific to peer-based learning in online communities. This
research will find out which elements in which schemas can be reused in order to make a
useful conceptualization of reputation that supports peer-based learning. Reusing schemas and
ontologies improve the likeliness of acceptance and prevent redundancy[81,82].

7
1.4 Problem statement 
With cheap or freely available technologies to create, disseminate, and evaluate information,
as well as social networks and specialized communities, the Internet is considered highly
suitable for active and networked learning. Whereas there are models in place to ensure
supply of learning resources and even guided courses free of charge, there is no model for
‘open’ assessment and accreditation of achievements in online communities. An online
reputation system that assesses learning (including problem-solving and teaching) in online
communities can be used to improve trust in expertise and capabilities of individuals based on
their actions and achievements online. Such a system needs to be developed in order to
improve access to the whole spectrum of education (Art.26 – UN Declaration of Human
Rights); from learning resources, to guidance, to assessment, to accreditation.[83]

This research will develop a reputation system to support the assessment of learning
achievements in online communities. The reputation system will take into account newer
learning approaches and intends to model the ‘value’ of objects (including persons) to support
learning. The solution may be inspired by various online initiatives that have a similar focus:
increase trust in objects by means of measuring social interactions with those objects. These
systems, one of which is Google, have not only proven to be very useful, but also
commercially sustainable.

A system that is able to recognize and interpret peer-assessment in learning communities can
be used to accredit, in some way, the achievements of an individual learner, and prove his/her
value. The direction we take is aimed at modeling learning interactions and knowledge
exchange in online communities.

The assumption of the research is that interactions and knowledge exchange activities can be
logged, and that these interactions contain information about someone’s expertise and value
in online communities. This information can be used in a professional context, if a member
chooses to share his/her reputation information with relevant third parties, such as potential
employers. Recognition by 3rd parties is an important incentive for individuals to participate
and interact in a way that increases their reputation.

8
 Interaction 

Motivation  SUSTAINABILITY  Reputation 

Recognition 

1.5 Outline of the proposal 
The following chapter is the literature review, which explains the above reasoning in more
detail.

• The first section will focus on the relevance of peer-based learning in online communities.
It concludes with a number of challenges, and an argument for the development of a
standardized reputation system aimed at peer-based learning.
• Reputation systems, focus of the second part of the literature review, have been developed
to increase trust in and quality of objects, improve interaction and support transactions
between people, and as an incentive to be active in a specified way. They can therefore
provide important lessons for the development of a scalable and sustainable system that
needs to do the following: assess the quality of learning resources, show the value (and
expertise) of individual people, and sustain ‘good’ or sustainable behavior in peer-based
online communities. Using exemplary reputation systems, it concludes by stating that
interoperability and openness of the reputation system will increase the likeliness of
adoption.
• Developments on semantic web technologies have created a useful framework to start
from, and that much of the work in this domain can be reused for the purpose of this
research. The final chapter is aimed at explaining the Semantic Web, and defines possible
trajectories that can be taken in order to create a useful, and accepted model for reputation
on the Internet.
The third chapter will describe the research philosophy, and defines the research questions.
The last chapter consists of a detailed planning of the research.

9
2 Literature review
In the literature review, each main section starts with an argument why the section is relevant.
The section ends with a few lines summarizing the conclusions and relevancy.

The literature review connects learning theories and newer opportunities enabled by the Web,
with motivation and sustainability. This is followed by an extensive analysis of reputation,
being an important incentive for knowledge sharing, and thereby a potential source for
sustainability of online peer-based learning environments. The relevance of the Semantic Web
in creating a valid and scalable reputation system is explained.

10
2.1 Learning 
The Internet and the numerous online communities of practice and professional networks
provide opportunities for informal, self-regulated and networked learning. Above all, the open
character offers relatively cheap access for individual learners worldwide to connect with
people and find relevant content. The Internet is an environment in which skills can be
developed that are needed in a technology driven, and rapidly changing society[3]. The skills
that learners develop in regular education systems are different from those developed in peer-
based communities[26,29,84]. The first part of this chapter will focus on learning theories and
the relevance of the Internet for learning. Because the research is focused on improving
sustainability of peer-based learning in online communities, a basic understanding of learning
and pedagogies is needed. In addition, if we intend to sustain peer-based learning through
semantic modeling of learning activities, it is important to know which activities constitute
peer-based learning.

The second part of this chapter will focus on the issue of sustainability in open education
environments and online communities. Whereas we have systems in place, developed over
centuries, that sustain learning in traditional environments, the systems in place that sustain
learning in peer-based online communities are still very much in their infancy. The coming
decades, we will see the emergence of institutions and systems that sustain high-quality
learning in peer-based online communities[5]. In order to create sustainable online
environments for learners to engage in peer-based learning, we must recognize and address
both the opportunities and challenges facing us. Quality management, assessment and
recognition, and motivation to collaborate or to share information are just a few of the
challenges[10,12,18,44,85,86,87,88].

The final part of this chapter describes the role of social mechanisms, including reputation,
with respect to these challenges. It concludes with an argument that many of the challenges
for sustaining peer-based learning in online communities can be influenced with a reputation
system.

2.1.1 Learning theories and pedagogies  
Below, a brief overview of the development of learning theories is given, based on Bransford
et al. [2].

Drawing on the empiricist tradition, behaviorists conceptualized learning as a process of

forming connections between stimuli and responses. Motivation to learn was assumed to be
driven primarily by internal drives, such as hunger, and the availability of external forces,
such as rewards and punishments (e.g., [89,90]). A limitation of early behaviorism stemmed
from its focus on observable stimulus conditions and the behaviors associated with those
conditions. This orientation made it difficult to study such phenomena as understanding,
reasoning, and thinking—phenomena that are of paramount importance for education. Over
time, radical behaviorism gave way to a more moderate form of behaviorism that preserved
the scientific rigor of using behavior as data, but also allowed hypotheses about internal
"mental" states when these became necessary to explain various phenomena. In the late
1950s, the complexity of understanding humans and their environments became increasingly

11
apparent, and a new field emerged—cognitive science. From its inception, cognitive science
approached learning from a multidisciplinary perspective that included anthropology,
linguistics, philosophy, developmental psychology, computer science, neuroscience, and
several branches of psychology [91,92]. New experimental tools, methodologies, and ways of
postulating theories made it possible for scientists to begin serious study of mental
functioning: to test their theories rather than simply speculate about thinking and learning
(see, e.g., [92,93,94,95], and, in recent years, to develop insights into the importance of the
social and cultural contexts of learning (e.g., [96,97,98,99,100].

The social nature of learning 
Constructivism is the psychological theory, which argues that humans construct knowledge
and meaning from their experiences [101,102,103]. Constructivist educational theory focuses
on concept development and deep understanding, rather than behaviors or skills, as the goals
of instruction[104]. Personal development and deep understanding happens through the
construction of meaning by the learner self, not through transmission from one person (the
teacher) to another (the learner). The fundamental principle of constructivism is that learners
actively construct knowledge through interactions with their environment [105][106].
Therefore learners are viewed as constructing their own knowledge of the world.

For effective learning, knowledge should be uniquely constructed by people through play,
exploration and social discourse with others. Learning objectives presented in constructivist
learning environments should be firmly embedded in context, and should, at least in some
way, represent every day life situations. Learners should also accept responsibility for their
own learning and be self-motivated to explore different knowledge domains.[104]

The central point of social-constructivism is an individual's making meaning of knowledge

within a social context [101]. Learning as a social practice is well established and dialogue is
one of the corner stones of social constructivism. This makes online communities such
potentially effective places for learning. The interactions in online communities is being
maintained through a sense of community and social capital through information flow,
altruism, reciprocity, collective action, identities, and solidarity to support the development of
democracy[64,65,107,108]. These are central elements that need attention in an online social
learning context.

Social mechanisms that address internal cohesion and sense of community are important for
learning and overall sustainability of a social learning environment, but so are mechanisms
that impact interaction with the external environment[58], including reputation and
recognition.

Learning in communities 
The term ‘situated learning’ locates learning in the process of co-participation and in the field
of social interaction, not in the head of individuals to get an inter-subjective understanding
and meaning of something[97]. In communities, learning means moving from the peripheral
(lurking, being introduced into processes, people, etc) into the center (sharing expertise,
making decisions). Peripheral participants do not accumulate knowledge and skills but are

12
introduced in processes, routines, networks, relevant issues, and approaches within the
community. “The individual learner is not gaining a discrete body of abstract knowledge
which (s)he will then transport and reapply in later contexts. (…) There is no necessary
implication that a learner acquires mental representations that remain fixed thereafter, not
that the ‘lesson’ taught consists itself in a set of abstract representations” [109].

Edgar Dale developed the so-called “learning cone” or learning pyramid [110]. It shows the
retention level in different learning activities, ranging from reading (low retention), to
hearing, to seeing to saying and doing (high retention). Although the research has been
criticized [59], it remains a powerful metaphor for communicating the effectiveness of active
types of learning.

Learning as knowledge creation is seen as the epistemological foundation of CSCL,

Computer Supported Collaborative Learning. Paavola, Lipponen and Hakkarainen explain the
“knowledge-creation” metaphor of learning as follows; “Learning is seen as analogous to
processes of inquiry, especially to innovative processes of inquiry where something new is
created and the initial knowledge is either substantially enriched or significantly transformed
during the process”[111]. Hence, learning goes beyond the information given.

Since traditional models of distance learning have not inspired researchers and teachers to
develop innovative pedagogical practices, current research and development work in the field
has turned towards creating multi-faceted pedagogical practices, utilizing ICT, that can
support learners in their efforts to engage in deeper-level learning and interaction[84]. Allert
writes that in modern knowledge societies, there is a need for scenarios that focus on
collaborative processes of creating innovative knowledge[109]. This type of learning
comprises of open, ill-structured problem solving processes, focuses on communication and
collaboration.

Stahl emphasizes that meaning is collaboratively produced in a cultural context, embodied in

a physical or semantic artefact, and is situationally interpreted within a community or social
system[84]. He refers to learning as shared meaning making, which is not understood as a
psychological process which takes place in individuals' minds but as an "essentially social
activity that is conducted jointly - collaboratively -- by a community, rather than by
individuals who happen to be co-located". Meaning is not transferred from one thinker to
another, but is constructed.

Processes of knowledge construction and shared meaning making happen increasingly in

virtual environments, such as games, online communities and forums. CSCL aims at
supporting this type of learning through the design of powerful learning and communication
environments integrating collaborative learning and the use of ICT [112].

Learning with understanding 
Historically, education has focused more on memory than understanding. An emphasis on
understanding leads to one of the primary characteristics of current theories of learning: its
focus on the processes of knowing [101,103]. Humans are viewed as goal-directed agents
who actively seek information. They enter a learning process with a range of prior knowledge,

13
skills, beliefs, and concepts that significantly influence what they notice about the
environment and how they organize and interpret it[97,98]. This, clearly, can have both
positive and negative consequences for the learning process and their abilities to remember,
reason, solve problems, and acquire new knowledge. Effective learning environments,
effective support systems for learning, and effective teachers therefore take into account the
background of a learner.

Control over learning 
New developments in the science of learning also emphasize the importance of helping
people take control of their own learning. Since understanding is viewed as important, people
must learn to recognize when they understand and when they need more information.
Effective learning environment therefore focus on sense-making, self-assessment, and
reflection on what worked and what needs improving[4,48,84,112].

Networked learning  
Learning is becoming a lifelong, self-directed and collaborative effort, in which one engages
with people and finds resources online. Learning institutions should focus on supporting this
process, and guide students in assessing and evaluating knowledge they encounter online.
Leaders at learning institutions need to adopt a more inductive, collective pedagogy that takes
advantage of the collaborative and participative spirit of our era and the potential of the
internet to connect people, link information sources, and support creativity. Rather than
individual learning based on competition and hierarchy, is a more networked model of
learning preferred, because it allows learning from peers, and stimulates cooperation,
partnering, and mediation[18]. Veen, Lukosch and de Vries describe a pedagogical approach
for networked learning, presented below[113].

Figure 2 - Networked Learning Model (Veen, 2006)

The ingredients of the Networked Learning model can be seen in the above figure. There are
four complementary areas that play an important role in knowledge development. Each of the

14
elements that are connected to these areas is relevant for this development process in which
the technology is a major facilitator for processes of communication, information retrieval
and information sharing. These areas are: Profiling, Connectedness, Knowledge and Business
Development.

• ‘Profiling’ states that individual users should take ownership of their professional
development, ICT enabling them to do this through social software tools. A way for
teachers of profiling is act as a tutor (individual online support, for example in forums),
coach (general support on specific areas), or scaffold (provides handhelds for students to
bring them further), and instructor (writes instructions and manuals). Students can also
profile their presence as helper and peer-tutor, or as a critical but just evaluator of learning
materials.
• ‘Connectedness’ stands for the connection between people and people and resources. It
relates to social networks and the way interaction and human relations are relevant for
people to perform in communities. These communities are fluent; you can take part for
some time depending on the purpose of the community. Communities are based on peer
references and are not limited to office hours.
• ‘Knowledge’ is the area that defines content and information in the Network Learning
Model. This content is distributed and discontinuous, stored in databases. Learners have to
aggregate bits and pieces (modules) into a meaningful whole. They do this
collaboratively, sharing their expertise with others.
• ‘Business Development’ is the area that describes the major companies’ business goals,
what they offer and for what purpose. These goals are the reference framework within
which learning takes place, it provides the organizational context (e.g. Dept. of Physics,
Univ. of Thessaloniki).
Networked learning focuses on interconnectedness between people and between people and
resources [37,38,39,40]. Technology is used to integrate delivery of knowledge with
interaction, communication and application[114]. The earlier mentioned concept of
Communities of Practice [29] is integrated in Networked Learning, because learning practices
and social practices are interconnected, the learning practices emerge from participants rather
than be imposed by facilitators, learners are involved in concrete practical actions together,
learning is not designed, rather designed for, variation in levels of expertise can expand the
group’s learning, networked learning needs to support visits to “otherness”[111].

Connectivism  
Widely adopted learning theories behaviorism, cognitivism, and constructivism, and
combinations of them, do not sufficiently explain the effect of technology in our lives and
learning activities. George Siemens and Stephen Downes have attempted to explain learning
in a digital age by combining and enhancing different learning views, and developed
Connectivism[4,21,115]. An important distinction from social constructivism is the emphasis
on the fact that knowledge does not need to be internalized and emphasizes that learning also
happens outside a person’s mind. Siemens argues that in the Information Age the learning
process concerns activities such as synthesizing and recognizing patterns, meaning making,
and forming connections between specialized communities. Know-how and know-what is

15
supplemented with know-where as the understanding of where to find the knowledge needed.
Connectivism addresses learning outside the person, knowledge stored in databases or other
electronic information holders accessible through the Internet. It describes a form of
knowledge and a pedagogy based on the idea that knowledge is distributed across a network
of connections and that learning consists of the ability to construct and traverse those
networks. This implies a pedagogy that seeks to describe 'successful' networks, as identified
by their properties, such as diversity, autonomy, openness, and connectivity; and seeks to
describe the practices that lead to such networks, both in the individual and in society[21].
Connectivism extends the notion of learning as a personal, internal change [116] to a network
change: Non-human elements act as actors in the network and the medium itself is part of
wider networks. A network that is able to adapt and learn is much more valuable than a static
network; therefore, if a person is connected to the dynamic, adaptive network, even though no
internal process occurs, in this respect the learner is a better learner.

2.1.2 Learning scenarios 
As we can see, there is no one definition for learning. In the context of this proposal, learning
is considered active, participative, and networked. Below, three short scenarios describe how
learning can occur according to this learning paradigm.

• In the first scenario, we assume learning to be a social activity that happens in

communities, where learners take different roles, depending on context and expertise. In
that case, if it would be a perfectly organized, large, and diverse community, each person
would be able to find a more knowledgeable peer, who, in theory, would be willing to
guide his or her learning, because that is a learning activity as well (explaining to others).
• Another scenario would be that several persons, through social interaction and online
discussion, combine their expertise to create new knowledge. Through social interaction,
they collaboratively come up with answers to questions, or develop new ideas.
• Another scenario, in the Connectivism paradigm, could describe how a learner tunes his
network, consisting of peers and several tools, such that at any moment, he or she is able
to know-where to find the relevant knowledge. This requires tools to manage knowledge
and information streams, but also to find relevant knowledge outside your own network.
How can we design communities in order to sustain this learning behavior? How do we
motivate people to engage in learning activities? How do we support the learner that the
learning activity is in line with his or her background, such that the right person or the right
resource can be found?

All these scenarios emphasize the importance of finding the right person, the right tool, and
the right content at the moment you need it. Finding consists of two parts: searching and
storing. You may be a very good finder, but if something is not stored at the place where you
look for it, you will never find it. On the Internet, if a document is not linked, you will not
find it. Keywords can be used to specify the search, and help the search engine interpret your
search.

Finding is also related with storing. Organizations used to store documents in enormous filing
cabinets, using taxonomies and armies of librarians to make sense of the information being

16
stored. Most databases and computerized file systems still function in this way, static and not
able to cope with a lot of information. Google, the most successful search engine, does not
store information and websites in this way.

In chapters 2.3 & 2.4 (on reputation systems and the Semantic Web), a more elaborate
explanation is given on possibilities of improving the way information is defined and stored,
increasing the likeliness that it is used.

2.1.3 Conclusions 
Different approaches to learning give opportunities to make online communities self-
organizing and effective places for learning. There are some important requirements;

• Social interaction leads to learning, sense of community, and trust

• Problem solving and knowledge creation processes are central to both learning and the
creation of new open educational resources
• Learning is a networked activity, which includes people and technologies in the network.
The ability to find people just in time, when you need it, is an essential feature, which can
be learned, but also supported through technology.
• The ability to find resources, people, answers, depends on the way information is stored in
the network.
• In order to improve self-organization of learning in online communities, new ways of
storing and retrieving information must be developed, specifically created for these
communities.
• An online reputation is a data profile. Such a profile contains information about
someone’s contributions and interactions in online communities. This information may be
rich enough to predict the value of the person to solve a problem or teach someone.
Because persons are central in a learning process, such information needs to be stored in a
way that it can be found.
• Background information is very important in learning. As learners bring their personal
history, bias, cultural characteristics, and expertise to a learning arena, reputation
information that describes expertise can be used to improve the learning experience for
individual learners.
• Reputation can also be used in order to stimulate people to engage in learning processes.
If relevant third parties recognize reputation, then it forms an important incentive for
lifelong peer-based learning.
The new science on learning sees the Internet as an opportunity to support peer-based learning
in online communities. Several initiatives have been started recently, but sustainability is still
a big issue[6,7,8,13,117,118]. The next section will focus on the sustainability of open
learning communities.

17
2.2 Sustainability  
Although the meaning of sustainability depends on the context in which it is used, some
generic things can be said. The term is often used to refer to the ability of ecological systems
to be usefully productive. Sustainability links present with future, because choices made now
should not compromise the opportunities or possible benefits in the future[119]. A typical
example of the opposite of sustainability is illustrated by a recent decision by CITES not to
ban fishing on blue-fin tuna1. Through overfishing the blue-fin tuna population is driven to
extinction, ruining ecosystems, and taking away opportunities for future generations.
Principles of sustainable development include (i) dealing cautiously with risk, uncertainty and
irreversibility; (ii) integration of social and economic goals in policies and activities; (iii)
equal opportunity and community participation; (iv) a commitment to best practice; (v) the
principle of continuous improvement; and (vi) the need for good governance[120].

Sustainability does not concern solely ecological and environmental systems, but relates to
the continuity of economic, social, and institutional aspects of human society, as well as the
non-human environment. Economic sustainability relates to the extent to which end-users rely
on subsidies or financial inputs, institutional sustainability addresses the effect changes have
on the social structures and institutions [121].

2.2.1 Sustainability of Open Educational Resources  
With regards to sustaining OER projects, there is no direct relation with nature or ecology. In
this context, sustainability really focuses on economic and institutional sustainability. Wiley
(2006) defines sustainability of an OER project as the ongoing ability to meet the goals of the
project[8]. This implies (i) continuously being able to produce and share Open Educational
Resources; and (ii) to sustain the use and reuse of their Open Educational Resources by end-
users. In other words, it means the ability to produce, share, localize, and learn from open
educational resources. This does not necessarily depend on top-down funding, but peer
production and reuse of OER can create sustainability[117]. Wiley claims that sustainability
can be reached through the reduction of friction and decentralization, capturing intrinsic
motivation of individuals to contribute without financial recompense. He literally states that
decentralization means the active involvement of students. Decentralization happens through
the peer production of open resources and sharing them in P2P (peer-to-peer) networks;

“It seems to me that sustainability and scalability are problematic only when people rely on
others to do things for them. Scalability and sustainability happen more readily when people
do things for themselves. Centralizing open educational services is less scalable/ sustainable.
Decentralizing them is more scalable/ sustainable. Wikipedia has two employees and well
over a million articles in multiple languages. We need to learn this lesson if open education is
really going to reach out and bless the lives of people.”[7]

This resonates with other research on sustainability of OER projects. Koohang & Harman
(2007) describe OER communities of practice as a means for decentralization and better
scalability [6]. The sustainability of Connexions, a project that almost entirely depends on the

1
http://www.cbsnews.com/stories/2010/03/18/world/main6311436.shtml

18
voluntary efforts of individuals worldwide for the production of OER, has been investigated
as a case study[11]. The case study focuses and describes issues regarding motivation and
value propositions for end-users of the site. Benkler describes criteria for (decentralized) peer
production of OER and its positive relation with sustainability[117]. Schmidt & Surman state
that the focus should be much broader than just on content: the sustainability of OER depends
on the thriving of a whole OER-ecosystem[19]. For instance, the process of building and
nurturing peer production communities should be taken more seriously. More focus should be
on creating stronger communities of practice in open education. Stephen Downes follows the
same argument, saying that the centralized model uses more resources, and is likely to cost
more, but offers more control over quality and content[7]. This, in fact, does not necessarily
means lower quality. We have seen that in Open Source Software, and also for online
encyclopedias[122].

Reasons for sharing educational resources 
An important question to be answered is of course; why? What are the reasons for institutions
and individuals for spending resources for giving away something that used to be competitive
advantageous? Reasons for involvement of institutions include altruism (knowledge should be
available for all), publicity & marketing, improving peer-network, and search for new
business models using open educational resources. The most common reason for participation
of individuals is the access to the best available educational resources and to have more
flexible materials. The four most important barriers for involvement in production of open
content, according to the OECD/CERI study on Open Educational Resources, are a lack of (i)
time, (ii) reward system, (iii) skills, and (iv) a viable business model[10]. Hylén (2007)
describes three challenges of the OER movement: copyright issues, quality assurance, and
sustainability[12]. Since 2005, various highly useful open licenses have been developed, most
famously the different Creative Commons license. Regarding quality assurance, three
approaches are mentioned, which concern the institution as final responsible organ (closed
and centralized), quality assurance by means of peer review (open and centralized), and the
final (open and fully decentralized) approach, as discussed above.

Quality of open educational resources 
In an earlier paragraph, we defined learning as shared meaning making, how do we define the
quality of this process and its outcomes? Quality management is defined as the structures,
activities, and processes that are designed for planning, assuring, improving and/or evaluating
the quality of an institution, a product, or a service [123]. The last couple of decades,
significant attention has been given to quality management approaches in e-learning. Markus
Wirth provides an extensive account of various initiatives worldwide and concludes that there
is an immense amount of offerings worldwide for quality management in e-learning. He
concludes that most of these initiatives and frameworks address traditional certification and
accreditation, and rarely do include newer quality approaches that focus on more innovative
learning. He points out that a useful categorization and systematization of these approaches
should help in comparing the enormous variety of approaches. He also stresses the importance
of context, because different quality approaches come from, and are based on different
contexts. Leaving out this context makes the whole effort redundant. Furthermore, he

19
mentions various contributions in international journals and books that show considerable
reservations towards the feasibility of quality management in e-learning. Traditional quality
standards in higher education mislead institutions to imitate classical face to face trainings
instead of fostering innovative approaches in learning and the use of (multi)media.

Based on Wirth, this proposal suggests quality management in online peer-based learning
communities to be defined as “the structures, activities, and processes that are designed for
the creation, planning, assurance, improvement and/or evaluation of peer produced learning
materials and interactions in online communities that lead to shared understanding”.

More specifically, this means

• the creation of learning resources, the use of them, and the discussions about these
resources;
• recommendations learning resources, creation courses or sets of resources, and
interactions that lead to structured learning, such as guidance or mentoring;
• retrieving, storing, and analyzing feedback and understanding user learner interactions;
• assessment of learning activities and contributions, and possibly some form of
accreditation; and
• offering incentives, support and motivation to members of online communities engage in
peer-based learning activities, like guiding peers or adapting resources.

Open Source Assessment and Accreditation 
Online resources, including Open Educational Resources, are needed, and sharing information
on the Internet is almost the easiest thing in the world. What needs to be developed, are the
systems and business models that truly sustain all the processes of online, peer-based
learning, not just the creation and sharing of educational resources (which is a very effective
way of learning). The systems and business model must be able to motivate individuals and
organizations to participate in these processes.

The introduction describes several initiatives that have extended the notion of open education
to offer not only the resources for free, but the structures to learn as well, including support
(through peer evaluation and by use of networked technologies). In one case, over 2000
students participated in a networked, online learning experience using a multitude of media
and technologies. One important thing that struck me was that official recognition could only
be offered to only 1 percent of the students, and for a fee. Recognition of online activities
through a traditional, formal accreditation scheme is not scalable in the online context.

This is in line with Schmidt et al. (2009), who identifies key challenges of learning in open
education communities: recognition, accreditation, and assessment [16]. Recognition is the
acknowledgment of achievements and conveys approval by a person, group, or organization.
It can be done implicitly (for example, citing a paper) or explicitly (recommending a person
on LinkedIn). Accreditation is formal certification by a community, institution, or
organization who thereby explicitly state that the receiver meets the standards of the
accreditor. Accreditation applies to individuals as well as to institutions and to programs.
Assessment is the process of determining the characteristics of something or someone. For

20
learners, assessment means determining their individual knowledge and/or skills, which is a
necessary basis for recognition or accreditation[124,125].

Making an analogy with open source communities, Schmidt et al. describe four criteria for
assessment and accreditation in open online learning communities:

• Trust: If a community or group as a whole is not trusted, the quality and reputation of
individual resources and people will neither be trusted. It is important that the trust in an
online community should be made explicit, for example by showing the community’s
output.
• Relevance & Quality: The quality of an individual contribution and contributor relates
with the way the community uses this contribution.
• Scalability: “A one-on-one system in which a trusted professor manually certifies a
student does not scale well. A community-based reputation or voting system scales more
easily, but outsiders might question its reliability.”[16] Human online interactions with
content can be used to calculate and predict things like expertise and value, but these
calculations must be trusted in order to have value. Trust in the system as a whole is a
crucial requirement.
• Transparency: Different assessment methods must be in place to reduce bias. These
methods, involving both human and digital resources, must be transparent, because open
systems encourage inspection and improvements and lead to higher standards of
accountability.
What we can expect in an open system of assessment is that achievement will be in some way
'recognized' by a community. This removes assessment from the hands of 'experts' who
continue to 'measure' achievement. And it places assessment into the hands of the wider
community. Individuals will be accorded credentials only when people in the community
think they deserve to be accredited. Downes describes two mechanisms of such a system:

• A mechanism whereby a person's accomplishments may be displayed and observed.

• A mechanism that constitutes the actual recognition of those accomplishments.[126]

2.2.2 Sustainability of online communities 
The above descriptions argue that self-organization and peer-based learning in online
communities can become an important and effective mode for learning. Sustainability of self-
organizing online communities is a complex issue, that relates to a number of social and
technical factors[58]. Supporting people to create new communities has the potential to
improve communication and support sharing of critical information and knowledge. It also
aligns with newer organizational views: moving from command and control to more
competency-based virtual communities [127].

Despite the popularity of online social networking sites, most initiatives fail to reach
momentum and fade away shortly after inception, because individuals lack the motivation to
be active [128,129,130]. In many online communities and websites that rely on community
participation, the majority of the contributions are done by a very small percentage of
members[131].

21
Understanding of the factors that influence motivation of individuals in online knowledge
environments can be used to increase willingness to invest time and share knowledge. The
following sections summarize mechanisms that influence motivation of individuals in online
knowledge environments, and their willingness to invest time and share knowledge. There has
been a substantial amount of research on motivation to share knowledge and increase
participation in online communities. There are various descriptive and prescriptive
frameworks that relate to the design of social or community software [107,132], community
sustainability [133], and drivers of community participation and contribution
[25,65,132,134,135]. Many studies point out that social and technical factors need attention in
order to enable knowledge sharing [132,134,136,137,138]. Thus, sustainability of peer-based
learning community contains elements of technology and organization.

Reputation 
Reputation relates to the concept of online identity and trust and is a primary research focus in
Web science2. Overview of past actions and participant identification helps to create and
sustain trust relationships in communities [135]. Trust forms the basis of a relationship and is
one of the most important enablers of community participation [25] and sharing knowledge
[138]. Reputation can be used as a virtual currency and is a conduit for trust in online
environments. Reputation is also used as a basis for recommendations and connections
between people and content. Rheingold (1993) in his discussion of the WELL, lists the desire
for status or prestige as a key motivation of individuals' contributions to the group [139].
Especially in knowledge or information sharing communities, recognition, status and
reputation form an important driver for individuals to contribute [66,140]. In Maslow’s
hierarchy of needs, helping others and being recognized because of that satisfies a person’s
need for esteem [65]. Wasko and others apply theories of collective action to examine how
individual motivations and social capital influence knowledge contribution in electronic
network. They conclude that people contribute their knowledge when they perceive that it
enhances their professional reputations[64,141]. Increased recognition also supports identity
building and belonging [107]. In online communities, people tend to share more information,
when their contributions are visible to the community as a whole, and to the extent there is
some recognition or praise or encouragement [142]. Many communities have therefore
features that show the level of contribution of individuals in ranking or increased visibility.

In Open Source communities, programmers are motivated not only by intrinsic aspects, i.e.
engaging in an activity out of pure pleasure, but also have in mind the signaling knowledge to
potential employers of profit-oriented companies [143]. The motivation of managers in OSS
projects, as well as of programmers can be traced back to career plans, which makes the
reputation one has within a community so essential [144].

2.2.3 Conclusions on sustainability 
As we can see, sustainability is a complex issue that is a goal in itself, but never an end.
Sustainability requires continuous adaptation to be able to meet the requirements of an ever-

2
http://webscience.org/research/roadmap.htm

22
changing environment. The Internet is a great place for certain kinds of learning, but
challenges relating with sustainability and quality are paramount. Online communities are
complex social and technological environments, and the design of sustainable online
communities very difficult.

Sustaining peer-based learning in online communities concerns, amongst other things, (i) the
production of useful learning resources, (ii) quality assurance, (iii) assessment and feedback,
and (iv) recognition for learning activities (including those activities that really prove certain
skills, such as explaining stuff to other people).

We have seen various initiatives that are able to sustain the creation and sharing of high-
quality open educational resources, and some do even offer learning support. With regard to
quality assurance in open education, most experts agree that a centralized approach toward
quality assurance is less sustainable than a decentralized approach. Similarly, assessment and
accreditation of learning achievements and activities cannot be done in a sustainable way if it
is directed in a top-down structure. Siemens and Downes have shown that even in an online
course with 2200 participants, feedback and guidance could be offered by the network, rather
than the course managers [1]. A more decentralized approach for assessment involves
providing tools for peer-based feedback (ratings, comments etc.) and communication.

Effective peer-based social learning in online communities requires participation on many

different levels and involves many activities. Even though some elements are lost in online
interactions (compared with face-to-face interactions), effective online learning in online
communities can be sustained when enough people participate, consuming various roles in the
process. These roles include the aforementioned peer-feedback and assessment, but also
organization roles, aggregation of content, creation, etc. Most of the processes that relate to
these roles can be considered forms of learning. Even though “learning” itself is an important
incentive for people to participate, more incentives are needed to sustain an online
environment. Literature on social mechanisms and motivation in online communities show
that reputation and recognition are crucial in motivating people to be engaged in the process
and share or create knowledge. Therefore, reputation could be used as an instrument to
incentivize behavior that sustains the various processes of peer-based learning in online
communities, including production and aggregation of resources, peer assessment, and
providing guidance. Reputation can be used as recognition for someone’s activities and
contributions in an online community. We have identified a lack of systems and models that
support official recognition (accreditation) in peer-based online learning communities.

The next chapter will focus on reputation and quality assurance systems. It addresses some
typical reputation systems, and concludes with a research framework for reputation.

23
2.3 Reputation and trust 
In the previous chapter, we concluded with challenges for peer-based learning in online
communities. The Internet provides with numerous opportunities for active, self-regulated and
networked learning. It is a giant network of networks in which the people communicate. The
architecture and intuitive tools that have been built (especially the recent years) allow for
creation and sharing information in social and professional networks. The Internet has been
used to disseminate educational resources for free, and there have been projects that offer
support and guidance, as well as educational technology, for free, in addition to the resources.
It seems that scalable and sustainable models are being developed for massive online courses
by supporting students to reflect and assess contributions in a peer-based manner. We have
identified a lack of models and systems that support recognition of learning activities in peer-
to-peer communities.

In this chapter, we look at different approaches to learn lessons for the design of a valid
system that supports recognition and improves assessment in online communities. The need
for such systems, and for research in this direction, is made clear by Schmidt et al. (2009):

“..despite improvements in methodology, assessment practices have a tendency to focus on

easily quantifiable measurements rather than contextualized behaviors, dispositions, and
attitudes. For our open education accreditation model, we are interested in retaining the goal
in accreditation of accurately reflecting learning and skills to enable individuals and firms to
negotiate employment arrangements efficiently. However, we also acknowledge that the skills
needed in the 21st century are radically different from those tested and accredited in the past.
Open education communities have certain unique characteristics that are ideally suited to the
development and recognition of such new abilities in its individual members.”[16]

They further argue that digital portfolios, digital trails (what you leave behind on the web),
and aggregations of individual opinions and ratings are used to improve relevancy in online
learning environments. Reputation models that calculate trust can enhance and improve the
accuracy these environment. Kollock (1997) has shown that reputation is one of the
fundamental motivations of people to share and create knowledge in online communities [65].
In this proposal, we agree with Schmidt et al (2009), that…

“..a better understanding of indicators for knowledge and skills in open education
communities is needed. Such indicators would consider processes and describe types of
communication and interaction as well as behaviors within a community of learners.”[16]

2.3.1 Why reputation?  
Tuomi (2007) describes that the social importance of formal educational certificates is now
declining, as the capabilities, interests, and reputations of people can be directly evaluated
using information and communication technologies. "Instead of asking whether a job
candidate has a formal educational status, a potential employer can now review the
candidate’s actual track record, blog postings, and possibly e-portfolios."[41]

In some expertise areas, such as computer programming, employment opportunities often

depend on a track record that can be reconstructed by search engines and personal blogs. The

24
digital identities of persons now consist of their own representations of achievements and
experiences, as well as reputations that accumulate through the comments of others. Formal
educational certificates may be components in such digital representations of capabilities, but
their relative importance will diminish [41]. Labalme and Burton (2002) call this reputation
capital, and argue that such values can be carried through systems [145].

McLure-Wasko and Faraj (2005) isolated individual motivations and social capital
considerations as main influencers on knowledge sharing. They had found that people tend to
actively contribute to online communities when they perceive that this enhances their
professional reputations [64].

In an essay on the future of online learning (2008), Stephen Downes writes the following;
“What will emerge for learning institutions, as for most other services, is a system of
reputation management that is integrated into the search process. Recommender systems, as
such systems are now called, will employ pattern-matching software to find resource
providers for potential clients. (Herlocker, Konstan, Terveen, & Riedl, 2004) The software
will draw information from a wide range of other services, including information about the
institution that produced the resource. As we have seen, though, with search engine
optimization (SEO) and other attempts to mislead reputation systems, there will continue to
be a tension between the trust we put in such systems and the degree to which they can be
infiltrated or corrupted. (Wu, Goel, & Davison, 2006) Reputation systems based on data that
can’t be replicated or imitated will acquire the most trust, and these will most likely be based
on verifiable identity and interactions within social networks.”[68]

Because reputation and trust are highly interrelated topics, we will define them below, before
continuing this chapter.

2.3.2 Trust and reputation 
Trust is the confidence in or reliance on some quality or attribute of a person or thing, or the
truth of a statement. In fact, trust is often used interchangeably with related words like
credibility, confidence or reliability[146]. Trust is the basis for interpersonal interaction and
especially for cooperation in a social network. Merriam-Websters Collegiate Dictionary states
that trust is “..the assured reliance on the character, ability, strength, or truth of someone or
something.”3

Kinateder and Rothermel (2003), from an Artificial Intelligence perspective, define trust in an
entity as the belief that under certain circumstances this entity will perform in a certain way
[147]. Another definition of trust commonly found is the one of Diego Gambetta “... trust (or,
symmetrically, distrust) is a particular level of the subjective probability with which an agent
assesses that another agent or group of agents will perform a particular action...”[148]. Lik
Mui (2002) is adapting this definition slightly in emphasizing the importance of expectation
instead of working with probability: “a subjective expectation an agent has about another’s
future behavior based on the history of their encounters”[149].

3
http://www.merriam-webster.com/netdict/trust

25
Reputation systems are used to increase trust in persons or objects. The scientific research in
the area of computational mechanisms for trust and reputation in virtual societies is a recent
discipline oriented to increase the reliability and performance of electronic communities
[150]. Usually, reputation is based on actions or achievements done by individuals or groups.
Whereas trust in between agents can be defined as a subjective expectation an agent has about
future behavior [149], reputation has more global characteristics.

Why are reputation systems so important for fostering trust among strangers? Online
reputation systems are developed to obtain and maintain measures of trust between people,
and intend to offer incentives to behave in a certain way. When people interact with one
another over time, the history of past interactions informs them about their abilities and
dispositions. The expectation of reciprocity or retaliation in future interactions creates an
incentive for good behavior. An expectation that people will consider one another’s pasts in
future interactions constrains behavior in the present [151].

Game theoretic approaches to reputation are predominant nowadays, resulting in systems that
are likely to give good results in scenarios that concern simple interaction patterns between
human beings, such as those in online marketplaces. However, when the complexity of the
scenario increases, these models are not so good. They reduce trust and reputation simply to a
probability or perceived risk in decision-making [150]. This seems to be too restrictive in
scenarios where the complexity of the agents in terms of social relations and interaction is
high. Reputation cannot be understood as a "static attribute, rigidly codified as footprints of
social hierarchy" [152]. On the contrary, it has dynamic properties, because reputation
attribution is a socio-cognitive mechanism that takes root in communication processes. Both
the "reputed agent" and the "reputing agent" should be taken into consideration, and the
context and relevant processes in which trust is established.

2.3.3 Context, transitivity, and cross‐community reputation 
An important point regarding trust is the fact that trust is not transitive [153]. Just because X
trusts Y and Y trusts Z, doesn't necessarily mean that X would trust Z. The problem is usually
one of context. So X can’t be sure in which context Y trusts Z. For this, approaches have been
suggested to incorporate context or categories into trust systems [154]. To translate too rigidly
or literally from the sociological and psychological domains can be difficult and would lead to
high dimensionality. Therefore, a set of categories or contexts would need to be chosen, for
which the trust between peers could be transitive for individual categories and could be
applied inter-categorically where possible.

Golbeck and Hendler have proposed the reputation inference algorithm, to be used with
semantic web based social networks, founded on the Friend-Of-A-Friend (FOAF) vocabulary
[76]. The FOAF project defines a mechanism for describing people and who their connections
are. We extend that ontology by adding binary trust relations (trusts and distrusts). Golbeck
and Hendler focus on social networks, and provide an analysis of trust between people and
how this trust could be inferred. In this proposal, we focus more on the inference of trust from
object to the object’s author. Transitivity then means that someone’s trust in an object (i.e. a
paper) is inferred by the author/creator of that object. Many other trust and reputation systems

26
intend to provide a ‘global’ characteristic [76,155,156], but in the importance of context
requires a different approach: reputation not as a global but a local characteristic.

A reputation in one online community is usually not reused in another community [63]. For
example, a seller on eBay cannot bring use his/her reputation to another online marketplace.
The other way around, it is also impossible to ‘bring’ your reputation to eBay and replace or
merge it with your eBay reputation. The trust people have in eBay’s reputation system, and in
the company, may be affected if it becomes dependent on other systems, managed outside
eBay. Yet, there are considerable advantages for communities and individuals to share
reputation between communities[63]. Resnick et al. argue that limited distribution of feedback
decreases its effectiveness, because reputation (both the good and the bad) relates to only a
single online arena[151]. The main advantages of using cross community reputations (CCR)
are (i) leverage of reputation data from multiple communities; (ii) producing more accurate
recommendations; (iii) reputation accumulation: a user does not have to build a reputation
from scratch; (iv) users are able to maintain (global or community-specific) offline reputation
certificates; and (v) faster establishment of new virtual communities by importing reputation
data from related communities [157]. Also, the trade of reputations may lead to new
opportunities for communities as a reputation provider. A reputation provider for experts in
solar energy for example, may offer communities and their members to share reputations
across a network of solar energy companies, researchers, and institutes. For this to happen,
reputations need to be constructed in a way that allows for interoperability and
synchronization between communities. Kinateder & Rothermel (2003) propose a directed
graph of categories to support mapping of dependencies and relationships in different
reputation systems [147]. Related context increases the likeliness of having similar incentives
and ranking between communities, improving compatibility and thus the possibility of
reputation mapping. Berlanga et al. (2009) warn for privacy issues that will arise when
participants’ online identity and reputation are transferred from one community to another
[67].

There are two preconditions for cross-community reputation systems: ontology mapping and
trust relationships. First, a community should define and follow a reputation ontology. If the
most important reputation mechanisms and mappings have been described, elements of
different ontologies can be mapped and related. Secondly, there is a trust relationship between
communities. Concerning CCR, trust can be defined as the extent to which one community
relies on another community to provide reputation for members of both communities [157].
As trust depends on context, more contextually similar communities are more likely to be able
to share and agree upon ontologies. This is explained in more detail in the next chapter.

In the following section, we will look at various systems that address trust and reputation. The
choice of systems relates with the topics identified in the previous sections. Important
elements include (i) the motivation toward people to behave in a certain (positive) way, i.e. to
share knowledge; (ii) reputation as a conduit for trust in quality or future behavior; (iii) the
relevance of reputation in a professional context; (iv) the indirect nature of reputation; (v) the
contextual nature of reputation; and (vi) the typology or ontology of reputation.

27
The contextual nature of reputation means that trust in reputation is also contextual. The
proposed research describes how reputation of someone in one online community, may be
useful and trusted by relevant 3rd parties. The contextual element of reputation should
therefore be included in a generic reputation system. The first example of the series concerns
Google, which is not so much a reputation system as a trust system. The system is able to
provide relevant search results based on an inherently simple algorithm that is specifically
aimed at containing the dynamic nature of the Web. Other examples focus on other aspects
that are considered important, such as having a defined ontology for reputation (eBay), the
professional relevance of reputation (Guru), and the indirect nature of reputation
(StackOverflow, research community).

2.3.4 Examples of trust and reputation systems 
Reputation systems have been around since the beginning of the web, and form a crucial
element for trust in many online communities, such as eBay. In their new book “Creating
Web Reputation Systems”, Farmer and Glass (2010) offer a framework to describe a
reputation statement [158].

Figure 3 - Reputation framework (Farmer & Glass, 2010)

Farmer and Glass distinguish a source, a claim, and a target as the basic constituents of a
reputation statement. In the following sections we will use this framework to describe
different reputation systems. By doing that, we hope to identify characteristics, advantages
and disadvantages of online systems that assess the quality of objects (including persons).

Google PageRank ‐ Trust in online content  
Even though Google's search engine is not directly considered a trust or reputation system, it
is significant in this respect. If we define trust as the confidence in something or someone to
perform or deliver something, we can look at Google as a system that ensures the probability
of a result to deliver the result (answer) on the question (query) asked. Google has been added

28
to this list because it has proven to be able to deliver highly relevant search results for free,
facing a rapidly expanding and more dynamic Web, and even become highly profitable.

Google’s PageRank system calculates relevance of a web resource by looking at its relation
with other web resources. PageRank relies on the uniquely democratic nature of the web and
uses its vast link structure as an indicator of an individual page's value [159,160]. In essence,
Google interprets a link from page A to page B as a vote, by page A, for page B. Votes are
weighed according to the PageRank of the voter. The image below makes clear how this
works, in a simplified form: A’s link to B is assigned more weight than c’s link to B, because
of differences in popularity (PageRank). Next to popularity Google analyzes contextual
factors of webpages and their relations to allow semantic searching.

Figure 4 - Google PageRank

As we said earlier, finding concerns two things: searching and storing. On the ‘storing’ side,
Google accumulates information about a webpage or URI (Uniform Resource Identifier:
digital object) by looking at its relations with other URIs. The ‘votes’ are contextualized
through interpretation algorithms that analyze the content and the topics affiliated with a URI.
On the ‘searching’ side, in most cases, Google ‘knows’ about the background of the searcher,
and thus being able to provide better results than a search engine that does not know anything
about the person who does the search. Making an analogue to learning, we have seen the
same: learning happens more effectively when the background, expertise and cultural
elements of the learner are taken into consideration.

Another important element concerns the way in which Google interprets ‘quality’, or in this
case, relevance. By measuring static and dynamic relationships between URIs, Google is able
to define a notion of relevance of a webpage and knows how to connect this notion to specific
contexts. It does not rely on specialized committees who analyze and evaluate the relevance

29
of a webpage, but looks at ‘the behavior’ of the Web by looking at the relations between
URIs.

Hence, by looking at relationships between resources, it is possible to create a sustainable and

scalable trust system to determine the relative value of objects. Google’s focus is on relevancy
of websites, rather than people, but a similar approach can be taken toward people. A
webpage is not the same as a person, but Google PageRank approach is very relevant to
consider in the development of a reputation system.

If we look at Google PageRank from the perspective of the Farmer & Glass reputation
framework, we see pages voting for each other, and that each vote also contains weight and
context. This makes it so powerful and instructive, we should look at how people use and rate
(vote for) contributions within communities, and determine how this influences the value of
contributions. As with Google’s PageRank, we should be able to determine the weight and
context (which are interrelated) of each vote.

eBay – Trust as a basis for transactions 
The biggest online retailer would probably succumb under its own success if sellers were not
subject to the site’s reputation system. Sellers are evaluated by four criteria: “Item as
described”, “Communication”, “Shipping time”, and “Handling charges”. This is part of
eBay’s reputation ontology, which describes interactions and relations and (ranges of) values
that determine the reputation of an individual seller. Clearly, this reputation system is mainly
developed with the objective of guiding the behavior and increase trust on eBay. For sellers,
other ways for a seller to improve their trustworthiness is to sell more items and being labeled
as favourite seller by buyers. Even though the system works (eBay hosts millions of
transactions per day), it has its critics. An experiment by Resnick et al. (2006) examines the
value of eBay’s reputation system [161]. It was found that only 0.6% of all the ratings
provided by buyers and only 1.6% of all the ratings provided by sellers were negative, which
seems too low to reflect reality. The possible explanation they provided for the positive rating
bias was that positive ratings are a sort of exchange of courtesies, whereas negative ratings are
avoided because of fear of retaliation from the other party.

Figure 5 - eBay reputation

The above example shows some interesting issues for reputation systems.

30
• Direct, explicit feedback: As shown by Resnick et al. (2006), direct feedback can be
ambiguous [161]. Anonymity may solve this problem, but may on the other hand cause
problems as well.
• Implicit feedback: Google has shown with its Pagerank system that it is possible to
determine relevancy or quality based on implicit factors. eBay shows number of
transactions, as one of these implicit parameters of trust.
• History: Next to figures showing feedback, the sales history of an eBay seller can be
accessed, including all the remarks made by buyers.
• Years active: An eBay seller’s profile always discloses the number of years or months
active on eBay, which also is a strong indicator of trustworthiness: you will have to start
all over again when you misbehave.
• Reputation ontology: Describing the parameters that influence reputation can be very
important, and generic, such as in this example. First of all, it is a powerful instrument to
guide behavior on an online community. Secondly, explicitly stating it could result in
standardization in an industry and allow for integration and exchange of reputation data.

Guru.com – Reputation to get a job 
Malone & Laubacher (1998) coined the term ‘e-lance economy’, meaning an economy largely
based on temporary organizations of individuals that emerge and dissolve when business
opportunities arise and disappear, and where IT serves to link individual nodes [162]. An e-
lancer, or electronically connected freelancer, is described as entrepreneur, independent
contractor, freelancer, independent consultant, or contingent worker. This kind of self-
employment is becoming more mainstream, and provides a number of advantages. Being
employed by several employers at the same time creates more security, and opens more
windows of opportunity [163]. Online marketplaces, including Guru.com follow this idea, and
provide a place for individuals and organizations to find each other and employ or be
employed. The mechanism is similar to eBay but concerns the ability of people to do a
specific task. More than being a good seller, someone on these platforms can be an expert in
something. Reputation relates with the money earned (and other implicit parameters), and
direct feedback by employers. Freelancers can also provide a resume, portfolio of work, and
do standardized tests to prove certain basic skills. Another way to make yourself visible and
increase reputation is to answer questions on the forum. The image below shows how
freelancers are presented in Guru, showing the most important trust parameters, including
badges (community accreditation).

Figure 6 - Guru.com reputation

An initiative that is more focused on learning is called Myngle, which is a language-learning

platform. On it, people around the world act as teachers as well as a learners. You learn what
you need, and put it practice what you know as a teacher or apply your skills directly and earn
money doing that. We can draw the following conclusions:

31
• Responsibility: First of all, members of these communities behave responsible, because
their earnings depend on the trust that is conveyed through their reputation profiles. Again
we see that reputation is a powerful mechanism to guide behavior and to sustain a online
social platform for professionals environment.
• Ontology: Secondly, as with eBay, transactions are evaluated by both parties, where
feedback is given. A more extensive overview of the topics covered by the community is
also made clear.
• Portfolio/History: The history of employment and a portfolio is shown, giving employers
more insight into expertise and competencies.
• Expertise: On eBay you can become a trustworthy seller by always keeping your
promises. eBay’s context is selling goods. On professional online communities, trust is
also contextual. On Guru, freelancers can be hired for almost all disciplines, Myngle
teachers come from all over the world. Information about expertise on platforms is
disclosed by freelancers and teachers themselves and evaluated through employment. The
image below shows that the contextual information of an interaction/transaction is
specified. For example, if a teacher on Myngle offers lessons in Italian and Spanish, and
she is a native Spanish speaker and can hardly speak Italian, then het expertise profile
will, over time, show that as well.
• Standardized tests: Internal and external accreditation possibilities improve trust in the
system.

StackOverflow community – Question Answering 
StackOverflow is an online community for people discussing IT-related issues. On an
advanced forum, community members can ask questions and/or provide answers. The
answers receive votes by community, and the answers with the highest amount of votes is
chosen as the best answer. In addition to votes and answers, members can add tags to
questions and answers. These tags play an important role in the reputation of users. Part of
member profiles is dynamically updated with the tags of Q&A topics they participate in, as
shown in the picture below.

Figure 7 - StackOverflow

32
After analyzing the mechanics of the community and its reputation system, the following
interesting elements are added to the discussion:

• Combine context and value: The system is able to combine the contextual factors (tags
assigned to questions) with votes to make a dynamic representation of a person. A profile
consists of static information of a person (added by him or herself) and of dynamic
information that incorporates expertise. Hence, members that answer questions build up a
reputation that includes context.
• From sharing to employment: On this site, members of the StackOverflow community can
add their resume and improve their professional profile, but their contributions in the
community and their reputation are visible as well to 3rd parties. These are potential
employers, and is an important incentive for the members to answer questions, especially
questions that prove their expertise. Even though this seems unusual, it is a standard
procedure for software giant IBM to look at a person’s reputation in an open source
communities when employing someone.
• Sustaining lifelong learning: StackOverflow and similar communities show that sharing
information for free can lead to employment and improve professional reputation. Active
and self-directed learning is possible and sustained in decentralized communities, if the
right tools and mechanisms are in place. Reputation, which is crucial for the electronic
freelancer, depends on contributions. It is an important incentive to collaborate in order to
get business opportunities.

Academic publishing and reputation 
Despite criticisms on the Impact Factor it is still widely regarded as one of the most important
quality measurement and assessment systems in the scientific community [164]. The Impact
Factor is a measure reflecting the average number of citations to articles published in science
and social science journals. It is frequently used as a proxy for the relative importance of a
journal within its field, with journals with higher impact factors deemed to be more important
than those with lower ones. The Impact Factor is shown in relation with other elements in the
simplified figure below.

33
 Figure 8 - Academic publishing

Universities use the Impact Factor to assess the quality of research of its departments, and
allocate funds based on that assessment. Hence, the influence of the Impact Factor should not
be underestimated. The Impact Factor received a lot of criticism because of its focus on the
journal in which a paper is published, rather than the number of citations it receives
[165,166,167]. In reaction, several other initiatives have emerged, including the h-index, and
PLoS (Public Library of Science) Article Level Metrics, focusing more on individual impact
and citation level. Currently, debates in the blogosphere focus on how this information can be
improved with a social impact level, by measuring popularity and usage [168].

Despite the different approaches to academic value and relevance, we can extract some useful
elements:

• Weight of source: The Impact Factor of a journal influences the value of individual
papers. This is similar to PageRank that also assigns extra weight to links (votes) that
come from popular websites. Even though there are different approaches, we see that
sources in a reputation statement can carry a weight.
• Gaming the system: Self-citation is mentioned as a way to game the system, and to
increase the impact factor of a paper. This however, is not always the case [169].
Reputation systems that do not address these issues are less valid and therefore less
trusted. Google develops advanced analysis tools to determine whether or not a website is
gaming its PageRank system, including looking at usage (time on site, number of
clickthroughs, etc.). This is a way to use the Web community to manage quality, which is
a sustainable and scalable way to do it.

2.3.5 Conclusions on trust and reputation 
The examples show us a range of approaches toward quality and reputation management in
online communities. Google’s PageRank is a highly scalable system that addresses the
dynamics of the web, but is less able to distinguish between online content and the producers
of content (people): it does not analyze the relative value of a person to do a specific task or to

34
know the answer for a question. eBay has developed a rather simple but effective trust
mechanism by looking at sales history and allowing buyers to reflect on the purchase. Resnick
(2006) argues however, that the feedback is not always to be trusted, because the direct link
between the feedback and the buyer causes fear of retaliation when giving negative scores
[161]. On online marketplaces, such as Guru and Myngle, trust is described in terms of
employment history, feedback, and artifacts, such as a digital portfolio. StackOverflow, an
online community for IT-related issues, successfully links question-answering to an online
reputation that is accessed by relevant 3rd parties (employers in the IT sector).

A reputation statement 
By aggregating the results, a framework to research reputation in online communities is
created. A reputation statement, as follows from the above analysis, contains three
dimensions:

1. Value, expressing the relative level of trust, appreciation, or quality (i.e. "good" > 0.8 out
of 1.0)
2. Context, describing all the relevant context factors, agreed upon by community and
relating to learning and (especially) teaching activities. The context is described in a
community specific metadata standard. the behavior that supports sustainability (i.e.
"quality of feedback", "Ergonomics and RSI") > in the case of peer-based learning in
online communities, context will relate to the community domain and specific skills and
competencies that can be expressed through knowledge sharing and peer support
3. Weight, which is especially relevant in case of assessment of knowledge or quality of
highly specific information: expertise should be acknowledged. The weight has a dynamic
relationship with the context: the context determines the weight of a source with a
reputation statement.

The framework is shown in the picture below.

Figure 9 - Reputation scenario

The above picture shows a reputation statement from a person to an object. This is not the
only possible way in which ‘value’ or quality can be expressed in a reputation statement,

35
anything that can be identified on the Internet can be both a source object and a target object
in a reputation statement. Other possibilities are from one object to another (i.e. a citation in
one paper to another), or a direct recommendation from one person to another (i.e. LinkedIn
recommendations). In the above picture, a person makes a reputation statement about an
object, i.e. a rating of a document.

• The person is identified with a user ID, and has a reputation profile, described in
metadata. The source’s reputation profile can be used to influence the weight assigned to
the reputation statement. One could argue that if a person is a world expert in a certain
topic, he/she is better able to express an opinion (reputation statement) on that topic.
• The reputation statement has a certain type that can also influence the weight of it, i.e. a
rating for a document is more explicit than a single visit by a person, and may therefore
carry more weight.
• Weight is a calculated value, based on the metadata of the source object, the type of value
statement, and the metadata of the target object.
• The target object is identified and contains metadata. When the target object is not a
person, then the target object has one or more authors (persons), who will inherit
information from the reputation statement.
An important distinction must be made between person and object. Even though both can be
source as well as target object (as explained above), a person does not have an author. An
object, which can be an activity or a document is always affiliated with one or more persons.
When a target object receives a rating, the reputation of the object’s author(s) is automatically
changed. The argumentation for this logical consequence is explained by Resnick and others,
who claim that direct feedback may be ambiguous [151,161].

Another very important aspect in the picture is the reputation ontology. The reputation
ontology describes all the relevant elements for reputation profiles, including the most
relevant (knowledge sharing and learning) activities that should be monitored, and the types
of reputation statements. The ontology is something that is inherent to a community and is a
common agreement on how things work in a community. If ontologies of similar communities
are described in a similar way, it may be possible to use reputation data across communities,
improving its value.

The next chapter will focus entirely on the semantics of reputation, and ways to include
relevant semantics into a reputation profile. We have described the relevance of context in
reputation, and Semantic Web technologies, with standardized formats to annotate data, tools
to create, share, use and combine ontologies, could be used to standardize the way
information about achievements and contributions in peer-based learning communities are
analyzed and stored. The Semantic Web is the idea of having a URI (Uniform Resource
Identifier) for (each) word in a document. A URI is a link to a shared definition of the word.
These definitions are usually stored in specific (metadata) libraries or repositories, such as the
Dublin Core. The Semantic Web allows for expression of different types of relationships
between people, objects and concepts. By using standardized and shared ontologies
expressing individuals, profiles, social connections, and content, it provides a way to connect
people and objects in an interoperable and extensible way.

36
37
2.4 Semantic web 
The previous chapter shows an overview of trust and reputation systems on the Internet. A
variety of approaches have been developed, all semantically different from one another, even
though some of the systems share common features. In this chapter, we describe how the
Semantic Web can improve reputation systems by providing a standardized framework to
describe and annotate information on the Internet.

In computer science, semantics means 'unambiguous'. The Semantic Web thus means making
the web less ambiguous by defining rules for interpretation [170]. Reputation is a very
ambiguous term. A semantic standard for reputation in peer-based learning communities will
not make reputation less ambiguous, but allows communities to adopt their own definitions of
reputation in a standardized way. This will make interoperability of reputation profiles easier
and therefore more probable.

The proposed research will provide a standard model to assess the expertise of people based
on the quality of content and interactions in online peer-based learning communities. Such a
model describes, in a very abstract way, all the elements and relations that constitute
reputation statements and how these reputation statement influence reputation profiles. The
actual ‘calculation’ of reputation can be based on parameters defined in a community- or
domain specific ontology. For example, the semantic model may describe the abstract term
"reputation statement”. In a specific domain or community, specific vocabularies can be used
for these evidences. An academic community, for instance, may use an ontology describing
“citations” as important reputation statement, and “accepted_by_journal” as another important
one. The Public Library of Science has started an initiative that explicitly includes all kinds of
metrics, not only the Impact Factor of a journal [171]. A developed semantic model also
define the all the source and target objects that are able to cite, be cited, accept, and be
accepted.

2.4.1 What is the Semantic Web? 
The semantic web is a large-scale, decentralized endeavor with the objective of bringing logic
and meaning to the web. Currently, most web sites and resources on the web are described in
a way that they can be understood by humans, but not by other computers. Formal and
explicit agreement on meaning of concepts and the explicit translation into computer-readable
language helps in making relevant information useful for computers and applications, offering
a wide range of opportunities for calculation of relevance and making recommendations
[71,172,173,174,175]. The semantic web is the idea of having a URI (resource identifier: link)
for each word in a document. A URI is a link to a shared definition of the word. These
definitions are usually stored in specific (metadata) libraries or repositories [71].

The semantic web is made up of assertions, such as the following

• Thieme Hennis is the author of this document;

• The author of this document likes pancakes.
Now you can make up that Thieme Hennis likes pancakes. The semantic web specifies ways
of exposing these kinds of assertions on the Web, so that third parties can combine them to

38
discover things that are true but not specified directly. Using syllogisms can be useful, but
should be considered with care, Clay Shirky argues (2003):

"In the real world, we are usually operating with partial, inconclusive or context-sensitive
information."[176]

His critique mainly focuses on the fact that human beings talk in generalities, and that the real
complexity of the world cannot be modeled using semantic technologies. The semantic web
proponents do not argue that this will be done. The objective is to make agreements on how to
annotate data, and as Jim Hendler says: 'a little semantics goes a long way'. He means that
adding only a little semantic annotation may increase the value of an application or document
significantly [177].

The semantic web enables more effective communication, because with standard definitions
and rules for interpretation, tools can be build that improve searching and filtering
information, matching content and people, and allow for more decentralized ways of working
[173]. In the proposed research, existing frameworks and ontologies will be used to create a
semantic standard that models the way reputation statements flow through a peer-based
learning community.

Example: Semantic Search 
We see an interesting example of the potential of the Semantic Web when we compare the
search engines Google and Wolfram Alpha. In Google, if you type “What is the weather in
Amsterdam?”, it will show you a range of search results, including hotels and businesses in
Amsterdam, and different websites with weather forecasts. When we type in the same
question in Wolfram Alpha, we are not presented with a list of links we have to interpret
ourselves to find the weather in Amsterdam, but it shows us a page containing the current
temperature, humidity, and all other weather related metrics for today. It also shows the
weather during the whole year, using graphs and various metrics, and suggests additional
metrics. Wolfram Alpha is a semantic search engine, and knows exactly what “What is the
weather in Amsterdam” means. By using resources on the web that are also semantically
annotated, it can understand the question, and find the relevant information to present the
answer you are looking for. This is only possible when resources are semantically annotated.
It has to be noted that Google is embracing semantic web technologies, as do other numerous
other big and small companies, governments, and institutes around the globe. [178]

2.4.2 Semantic Web Technologies 
The core of the Semantic Web concerns the following technologies;

• Content on the web is annotated with custom XML tags, like this: <person>Thieme
Hennis</person>
• XML provides a syntax to encode data; the resource description framework is a
mechanism to tell something about data. As its name indicates, it is not a language but a
model for representing data about “things on the Web”. This type of data about data is
called metadata. The “things” are resources in RDF vocabulary.

39
• Relations are annotated in RDF triples; (Thieme Hennis)(is_engaged_to)(Sofie Blaisse).
Triples are
Ontologies describe the relations between concepts and define the concepts, including the
hierarchy; (person) has (father) & (mother). This is usually done within a viewpoint, meaning
that the ontology is useful within a certain context. Inference and reasoning rules describe
logical relationships, such as "If (person) is married, then (person) has (parents in law)".

2.4.3 Relevance of the Semantic Web 
So why is the Semantic Web an important domain that must be integrated in this research?

Except for the most fundamental and common values (Thou shalt not..), in most other cases
quality and expertise are concepts that have value only within a specified context. In the
proposed research, we look at interaction patterns and how quality or expertise can be derived
from these. Online behavior of people can be monitored and analyzed to provide a very rich
picture of the community and its members and resources. The analysis of the behavior and
patterns leads to profiles of people, resources, and communities as a whole.

A standard for analyzing and storing reputation information allows for exchange between
communities, and so data from different sources can be leveraged to improve trust and quality
of recommendations. Another advantage is the expansion of the network of resources and
people, being able to find relevant people and resources across networks and communities,
rather than within communities. In a growing knowledge economy, applications and tools that
enable searching and finding relevant persons across the Web will have high impact and
commercial value.

Semantic Web technologies improve both sides of the search-coin: storing and retrieving
information from large data sources. Because relations are explicitly stated, and concepts are
defined in unambiguous ways (if possible), interesting combinations and calculations can be
made. One of the imagined applications may be a system that is capable of describing the
value of someone based on activities online. If these activities are not explicitly described, the
context in which value emerges, cannot be attained by computers. Therefore, semantic
annotation of relevant concepts will improve the final artifact in terms of quality of
information stored in reputation profiles, trust through transparency, and ability to store
contextual information in a decentralized way. Also, Semantic Web technologies support the
creation of ontologies and other artifacts.

40
3 Scientific approach: philosophy, strategy,
and planning
Richard de Neufville describes four issues that need to be addressed in the formulation of a
research project [179]:

1. What is the question or issue?

2. What method will be used to address these questions?
3. What evidence can be applied?
4. What logic integrates the above?

Based on the problem statement, and the issues identified in the literature review, the
following section will describe the main research question and a number of operational
research questions. It further elaborates on the research philosophy, strategy and the required
research instruments. The planning of the research is also presented.

3.1 Research questions 
A thesis is helpfully framed as a question that defines the issue under consideration,
comparable with a hypothesis. Research questions must have more than one possible answer.
If a research question cannot be refuted it is a truism of no real interest [179].

Several researchers have shown the importance of reputation as a motivation to share

knowledge in online communities. The objective in the long run is to improve sustainability
and self-organization in peer-based learning communities, by means of an incentive that
continuously motivates people to behave according community standards. The core topic of
this research is thus reputation and its relation with learning activities in online communities.

The main research question is aimed at the design and evaluation of a reputation system to
provide a continuous incentive for individuals to share knowledge and engage in teaching and
learning support activities in online communities. On the one hand, reputation systems must
be designed such that it guides behavior in a way that it sustains and supports the most
important activities in a community. In learning communities, these consist of knowledge
creation and sharing, learning support and guidance, and feedback and assessment. The nature
of these activities provides us with an opportunity not only to model a community ‘value’ or
trust, but specific expertise and competencies. Expertise and competencies are expressed
through a published paper, providing useful feedback, or some kind of learning support or
guiding activity. The content that is produced, interactions in which knowledge is shared, and
peer-guidance activities and teaching are all things that can be monitored and analyzed to
create expertise profiles of individuals.

The proposed research system, therefore, must be able to recognize relevant activities and
reputation statements, and develop reputation profiles that can be used to determine
someone’s expertise or knowledge. The main research question is:

41
• How does a scalable and trusted online reputation system look like that creates expert
profiles from learning support, feedback, teaching, and knowledge-sharing activities in a
peer-based, networked learning environment?
Scalability is important, because that is one element we have identified in the problem
statement: traditional approaches to assessment and accreditation (a form of reputation) of
individuals is not scalable to the online context; only a very limited amount of people can be
assessed for credit. Trust is another essential element, and concerns the fact that reputation,
when trusted across communities and especially by 3rd parties (including potential
employers), becomes a strong motivation. Finally, the reputation system must ensure that
activities are monitored that sustain and support peer-based learning. In this research, the
nature of these activities allow for expert profiling.

3.1.1 Operational research questions 
The sub-research questions focus on these four elements: activities that sustain peer-based
learning, the modeling of expertise or value based on these activities, scalability, and trust in
the reputation profiles.

Defining peer‐based learning 
First, we have to define peer-based learning in online communities, and develop an ontology
describing useful elements, relations, and concepts that, as a whole, constitute knowledge
sharing and learning support. The research is less focused on motivating learners to consume
information than motivating people to share knowledge (to learn from) and to offer support to
people who want to learn.

• How can knowledge sharing and learning-support in peer-based online communities be

characterized?

Inference of knowledge 
Next, we must analyze the relation between knowledge sharing activities and learning-support
and knowledge and competencies. We have seen that inference of attributes can be difficult, if
context is not addressed.

• How can individual knowledge and competencies be inferred from shared knowledge and
learning support activities?

Scalability 
The third requirement concerns scalability. Scalability in this research means the ability of the
system to produce accurate reputation profiles with increasing numbers of users. Another
dimension of the scalability is its adoption rate: can the reputation system easily be adopted
by learning communities and remain useful? As we indicated earlier, the Semantic Web and
its technologies are a basis to work from.

• How can Semantic Web technologies increase scalability of the reputation system?

42
Trust  
Finally, the produced reputation may be scalable, but if it is not trusted, then it remains
useless. Trust has two elements: (i) trust by the reputed agent that personal information is not
shared with 3rd parties without consent; and (ii) trust by 3rd parties in the produced
reputations. The first concerns transparent and explicit rules about ownership and control. The
latter is merely concerned with the quality of the reputation profiles. The two perspectives do
not have opposing concerns, but there may be a tender balance between privacy and control
on the one hand, and quality of the profiles on the other.

• How can trust in the reputation system be ensured from the perspective of the reputed
agent as well as the interested 3rd party?
The objective of the research is to create a system that can be used to communicate the value
an individual brings to a learning community. Because learning is a social activity, in which
meaning is constructed, learning activities could include blogging, question answering,
editing a wiki page, and publishing content (including interactive media). It is not the
intention of the research to describe in detail all these activities, because they may be different
for each community. What is the intention though, is creating a generic system that is able to
adopt specific descriptions and ontologies of peer-based, online learning. The system must be
flexible enough to be tuned to specific requirements set by one community, meanwhile
adhering to the same standard as other communities to make reputation shareable and trusted
in a wider context.

The product of the research constitutes of a reputation system that is defined according to
semantic web standards, a methodology to design reputation systems for peer-based learning
communities, and a report. Below, the scientific and societal relevance are defined.

3.2 Societal and scientific relevance 
This year’s Pew Internet survey on the impact of the Internet on institutions in the future
describes the following:

"The respondents who addressed the issue of “innovative forms of online cooperation”
sometimes referred to activities between people and institutions that were post-bureaucratic.
They argued that people could use the Internet and cell phones to create alternative, un-
bureaucratic structures to solve problems through network-structured communities."[180]

One of the important characteristics of these innovative ways of learning and collaboration is
that it is networked: you should be able to find the relevant person in various online networks.
Repeating the earlier statement that finding involves storing and searching, we can argue that,
if online communities and professional networks maintain their own standard for the storage
of user profiles, this vision is not likely to become truth. Standardization and interoperability
improves the Internet as a place for collaborative, networked learning.

A standardized and interoperable reputation system improves the sustainability of peer-based

learning communities. This increases access to all elements of education, even professional
guidance, to anyone with Internet access. It also supports electronic freelancers in profiling

43
themselves, and innovative and networked structures (with reputation/identity providers) for
work may emerge from the learning communities.

Not only online communities may benefit from such a standard, but traditional educational
institutes as well. By adopting the standard, they might be able to decentralize a part of their
efforts in education, and focus on specific activities that cannot be done online. They may
even partner with trusted online communities in providing accreditation for reputation, or
engage in other innovative collaborations.

Finally, in a broad sense, the reputation system may be applicable in a much wider context
than only the peer-based learning communities. In organizations, for example, the same
reasoning may be applied to find experts, and provide an incentive for the people to share
knowledge or comment on shared information.

Specific scientific contributions are the improvement of ontologies, algorithms and

technologies that relate with expert modeling. For education science, this research may
provide with a useful model and technology to support for peer-assessment and measure and
describe knowledge and competencies.

3.3 Research philosophy and methodology 
Research philosophy defines a view on reality and a view on knowledge giving us the ability
to choose an appropriate method to conduct meaningful and valid research. Research
philosophy is built on an ontological approach (the way the nature of reality is viewed), a
theory of knowledge acquisition (epistemology), and the practical approach to knowledge
acquisition (methodology). Epistemology and methodology are intimately related: the former
involves the philosophy of how we come to know the world and the latter involves the
practice. Validity is guaranteed through rigorous and correct application of the research
methodology. Mainly based on the ontology, epistemology and methodology we can
differentiate among different paradigms. A paradigm in research philosophy defines what
falls within and outside the limits of legitimate inquiry. It also states that different
assumptions concerning in paradigms “cannot be dismissed as mere “philosophical”
differences; implicitly or explicitly, these positions have important consequences for the
practical conduct of inquire” [181]. In scientific literature, 4 paradigms are identified:
positivism, post-positivism, critical theory and constructivism [181,182].

Positivism is historically the oldest paradigm dominating natural and social sciences for
centuries. Positivism states that reality is driven by natural laws and mechanisms, which can
be comprehended and studied. The inquirer should not influence his object of study and
should therefore take the necessary measures to avoid this. As such he should be capable of
formulating an almost exact representation (e.g. a set of definitions for the laws) of this
reality. Post-positivism was born as a criticism on positivism. It tries to be less certain about
claims about reality. It is still close related to positivism and states an objective reality but the
apprehension of this reality can only be imperfect and incomplete. Reality can only be known
approximately, thus without every fully know it. Critical theory functions as an umbrella
paradigm for several alternative paradigms. The common assumption of all these paradigms is

44
value-depended nature of inquiry. Reality is generally viewed as something shaped by
different values, e.g. economical and social. The inquirer is part of this reality and as such
influences the inquiry by its own values. Constructivism is finally a paradigm stating all
knowledge is constructed and depends on convention, human perception and social
experience. The inquirer and his object of study are strongly linked as the findings are created
as the inquiry proceeds. A way to understand these paradigms is to analyze them from the
ontological, epistemological and methodological point of view. An overview is structured in
the table below with a column for each paradigm and a row for each point of view.

Positivism Post-positivism Critical Theory Constructivism

Ontology Naïve realism, “real” Critical realism, Historical Relativism, local

reality but “real” reality but realism, virtual and specific
apprehendable only imperfectly reality shaped by constructed
and social, political, realities
probabilistically cultural,
apprehendable economic, ethnic,
and gender
values,
crystallized over
time

Epistemology Dualist/ objectivist, Modified dualist/ Transactional/ Transactional/

findings true objectivist, subjectivist, value subjectivist,
critical tradition/ mediated findings created findings
community,
findings probably
true

Methodology Experimental/ Modified Dialogic/ Hermeneutical/

manipulative, experimental / dialectical dialectical
verification of manipulative,
hypotheses, chiefly critical
quantitative methods multiplism,
falsification of
hypotheses, may
include
qualitative
methods

Tabel 1 - A comparison of inquiry paradigms in terms of ontology, epistemology, and methodology [181]

Our research subject is in essence a ‘wicked problem’, as defined by Hevner et al. (2004)
[183]. That is, our problem is characterized by:

• Unstable requirements and constraints based upon ill-defined environmental contexts.

• Complex interactions among subcomponents of the problem and its solution.
• Inherent flexibility to change design processes as well as design artifacts.
• A critical dependence upon human cognitive abilities to produce effective solutions.

45
• A critical dependence upon human social abilities to produce effective solutions.
This leads us to frame our research as both inductive and explorative. We are explicitly not
testing existing theories on learning or trust, although we are using theories to improve
developed artifacts. What we aim to do in this research is build theory and two artifacts (a
reputation system and a semantic model) that enable new forms of collaboration and learning.
We approach the research through inductive reasoning, which begins with specific
observations and measures in order to detect patterns and regularities, formulate tentative
hypotheses that can be explored, and finally end up developing a general conclusion or
theory. Through the analysis of several online communities, some generic characteristics can
be formulated, and described in an ontology for peer-based learning in online communities.
This ontology forms the input for the reputation system.

3.3.1 Ontology and epistemology 
Flood, in his synthesis of critical social theory and systems thinking, describes five onto-
epistemological views and their primary concerns [184,185]. He describes five onto-
epistemological viewpoints, capturing the subjective and objective extremes of ontology
(realist versus nominalist) and epistemology (positivist versus anti-positivist).

As we discussed above, the research aims to develop theory and create two artifacts (a
reputation system and a semantic model). These artifacts are created through the use of a clear
understanding of the relation between learning theories, reputation, and motivation, using an
accepted classification framework for the description of relations and objects on the Internet.
As the research focuses on a social-technical problem, realist ontology is appropriate. Realism
is the theory that a reality exists outside of us, and independent observers therefore can agree
upon that reality.

Post-positivism states that facts are fluid and elusive, forcing us to only focus on our
observational claims. Reality can only be known approximately; we cannot ever fully know it.
Thus, validity of knowledge of social reality is of low resolution, but with a view to working
on increasing validity.

We therefore argue that we can position our research within Flood’s “Distant Ontologist”
(Type II) onto-epistemological viewpoint, which means that validity and methodology are our
main concerns for our research approach. Facts about reality can be derived from the
investigation, and can be used to develop a theory and artifacts.

3.3.2 Methodology 
The research philosophy guides us to the research methodology. Methodology is the practical
approach for knowledge acquisition. Methodology is focused on the specific ways – the
methods – that can be used to increase understanding about the world. There are two
recognized methods: the quantitative method and the qualitative method. Qualitative methods
include the case study, phenomenology, grounded theory, and ethnography, among others.
Quantitative methods include hypothesis testing, power analysis, met analysis, observational
studies, re sampling, randomized controlled trials, regression analysis, multilevel modeling,
and high-dimensional data analysis, among others. The proposed research is aimed at

46
developing a semantic model as well as a reputation system. This inclines that qualitative
methods are in place to improve understanding about what constitutes reputation in the
defined context, and to develop the semantic model and reputation system. Quantitative
methods can then be used to test the scalability and usefulness of the system with large
numbers of users.

Problem‐solving process 
Taking the notions of ‘wicked problems’ and of bounded rationality as a starting-point, we
will describe the process of problem solving, based upon work by Mitroff (1974), and
presented by Sol (1982) [186,187]. Mitroff’s process consists of six activities and four stages.
The cycle starts with the problem as being perceived in reality; e.g. by a problem owner.
Through a task of conceptualization a conceptual model is constructed, which defines the
variables that will be used to specify the nature of the problem in broad terms. This
conceptual model, according to Sol, is formulated through the choice of a vehicle of
communication, a worldview, and a construct paradigm. The empirical model specifies the
conceptual model in terms of the system under study, and can be checked against the problem
as it was originally perceived; the correspondence check. With the empirical model solutions
can be generated and analyzed. Solutions must be checked on consistency with respect to the
conceptual model. After the choice of one solution, it will be implemented in the real world,
which can form the beginning of a new cycle.

Figure 10 - The problem-solving process (based on Mitroff 1974) [187]

3.3.3 Summary of research philosophy 
In summary, we set out to do inductive, explorative research. We have realism as our
ontology, with post-positivism as our epistemology, and the qualitative method as our
methodology. The proposed research, with the objective of designing a semantic model and

47
reputation system, leads to design science as our research approach. In the design science
paradigm knowledge and understanding of a problem domain and its solution are achieved in
the building and application of the designed artifact [183,188]. We argue that design science
is a good fit for Flood’s Distant Ontologist (Type II), as design science’s focus on rigor and
relevance fulfill the Distant Ontologist‘s need for validity and correct methodology.

3.4 Methodology: Design Science 
In the broadest terms there has been consensus: that design science research in information
systems involves learning through the act of building [189]. According to Hevner, March et al
(2004), two paradigms characterize much of the research in the Information Systems
discipline: behavioral science and design science [183]. The behavioral science paradigm
seeks to develop and verify theories that explain or predict human or organizational behavior.
The design-science paradigm seeks to extend the boundaries of human and organizational
capabilities by creating new and innovative artifacts. As Hevner, March et al consider both
paradigms foundational to the IS discipline, they use both paradigms to describe the
performance of design-science research in Information Systems via a concise conceptual
framework and clear guidelines for understanding, executing, and evaluating the research.

The design-science paradigm has its roots in engineering and the sciences of the artificial
[92]. It is fundamentally a problem-solving paradigm. It seeks to create innovations that
define the ideas, practices, technical capabilities, and products through which the analysis,
design, implementation, and use of information systems can be effectively and efficiently
accomplished. [183]

Hevner, March et al consider designing useful artifacts complex, due to the need for creative
advances in domain areas in which existing theory is often insufficient. They broadly define
IT artifacts as constructs (vocabulary and symbols), models (abstractions and representations),
methods (algorithms and practices), and instantiations (implemented and prototype systems).

Design science creates and evaluates IT artifacts intended to solve identified organizational
problems.’ These are assessed through evaluation methods that are scientifically valid and
create the opportunity to refine the aforementioned theories and methods. Rigor is derived
from the effective use of the knowledge base – theoretical foundations and research
methodologies. To not loose sight on the applicability of the research in real world
environments, relevance is achieved by conducting research that addresses business needs.
The output of the research is material useable in these environments. Success is predicated on
the researcher’s skilled selection of appropriate techniques to develop or construct a theory or
artifact and the selection of appropriate means to justify the theory or evaluate the artifact.
[183]

The further evaluation of a new artifact in a given organizational context affords the
opportunity to apply empirical and qualitative methods. The rich phenomena that emerge
from the interaction of people, organizations, and technology may need to be qualitatively
assessed to yield an understanding of the phenomena adequate for theory development or
problem solving. [190]

48
The design process, used in design science, is a sequence of expert activities that produces the
design artifact, as depicted in figure below. The evaluation of the artifact then provides
feedback information and a better understanding of the problem in order to improve both the
quality of the product and the design process. [183]

IS research is conducted in two complementary phases. Behavioral science addresses

research through the development and justification of theories that explain or predict
phenomena related to the identified business need. Design science addresses research
through the building and evaluation of artifacts designed to meet the identified business need.
The goal of behavioral- science research is truth. The goal of design-science research is
utility. [183]

Hevner describes three similar cycles that make Design Science research rigorous and
relevant, see picture below. These are the Design cycle, the Relevancy cycle, and the Rigor
cycle. It is not uncommon for a researcher to address all cycles at the same time.

Figure 11 - Information Systems Research Framework [183]

Design science is especially suitable for the development of ‘supporting tools’; e.g. methods,
tools, approaches that can be implemented in organizations or systems to improve the ways
people are working. When complex solutions have to be designed rigorously, while remaining
relevant for stakeholders, design science offers a suitable framework for research [191].

3.4.1 Guidelines  
Design science is inherently a problem solving process. To help in this process, Hevner,
March et al (2004) have derived seven guidelines, based on the fundamental principle of

49
design-science research: that knowledge and understanding of a design problem and its
solution are acquired in the building and application of an artifact[183].

Guideline
1. Design as an artifact
2. Problem relevance
3. Design evaluation
4. Research contributions
5. Research rigor
6. Design as a search process
7. Communication of the
research
Tabel 2 - Design Science Research guidelines [183]
Description
Design-science research must produce a viable artifact in the
form of a construct, a model, a method, or an instantiation.
The objective of design-science research is to develop
technology-based solutions to important and relevant business
problems.
The utility, quality, and efficacy of a design artifact must be
rigorously demonstrated via well-executed evaluation methods.
Effective design-science research must provide clear and
verifiable contributions in the areas of the design artifact, design
foundations, and / or design methodologies.
Design-science research relies upon the application of rigorous
methods in both the construction and evaluation of the design
artifact.
The search for an effective artifact requires utilizing available
means to reach desired ends while satisfying laws in the problem
environment.
Design-science research must be presented effectively both to
technology-oriented as well as management-oriented audiences.

3.4.2 Design Science activities 
Artifacts constructed in Design Science research are rarely full-grown information systems
that are used in practice. More often, artifacts are innovative concepts that define the ideas,
practices, technical capabilities, and products through which the analysis, design,
implementation, and use of information systems can be accomplished. Kuechler and
Vaishnavi (2008) regard design theory as a prescriptive statement, a significant, possibly the
most significant, output of the research effort [189]. Venable (2006) created a framework for
Design Science, describing design science activities and theory development [192], based on
Kuechler and Vaishnavi (2008). For our research, we adopt this framework, which is shown
below, as the outlook on design science as a research strategy.

50
 Figure 12 - Design Science activity framework [192]

3.5 Design cycles and instruments  
Our chosen research strategy provides us with the steps that need to be taken in our research,
and also the research instruments that are needed to support our research strategy. Following
both Sol’s process of problem solving (Figure 10), and Venable’s activity framework (Figure
12), we will use three of the Venable’s design science activities (Problem Diagnosis,
Technology Invention / Design, Technology Evaluation) in a sequential design cycle. All
these activities actively contribute to the fourth activity: Theory Building (e.g. problem
theories and solutions). Our research planning will utilize two of these design cycles: one for
developing the semantic model of reputation, and one for designing the reputation system.
Thus, in our research we will start out with gaining an increased understanding of the
problem, as perceived in reality. Based on this analysis, a conceptual model of our technology
design will be made. This model will be evaluated, resulting in an empirical model. All these
steps, and their results, contribute towards theory building and finding a solution for the
original research problem. This approach translates to two research cycles, each comprised of
three steps: problem diagnosis, technology invention/design, and technology evaluation.

3.5.1 First cycle: developing the semantic model 
The first cycle is aimed at developing the semantic representation of reputation in peer-based
learning communities. This model of reputation intends to improve motivation, by making a
representation of someone’s value or expertise, based on knowledge-sharing, learning

51
support, and teaching activities. The model adheres to semantic web standards, for reasons
explained in chapter 2.4. The specific steps are defined below:

Problem diagnosis 
In this step, we have to describe typical activities of knowledge-sharing, learning support, and
teaching activities in online communities. Relevant activities are activities in which
knowledge is created or shared, evaluated, commented on, and happen in a peer-2-peer
manner.

• Literature on active and self-organized learning will form the initial input for describing
the generic characteristics of the learning, and create an ontology for this type of learning.
• In addition, the online community Peer 2 Peer University4 will be analyzed using social
network analysis and data mining in order to derive generic elements for the conceptual
model.
With regards to theory building, this research step results in an overview of incentives for
offering peer-support, and a description of typical activities that sustain learning in a peer-
based online community.

Technology invention / design 
The design phase is aimed at translating the conceptual model of peer-based learning and
teaching, and the relevant aspects that incentivize behavior, into a semantically correct model,
abstract enough to be implemented in various communities, and flexible enough to adopt
existing community-specific ontologies. Adopting the Web Ontology Language, the Resource
Description Framework, and other semantic standards, a semantic representation will be made
of the conceptual model. In order to improve acceptance and prevent redundance, if possible,
existing elements are used. For instance, the model could be developed as an extension of the
widely adopted FOAF (Friend-of-a-Friend)5 standard.

Evaluation 
The developed semantic model of reputation (as defined in this research) will be shared
online, using my personal wiki6. Next to making every step visible and available, important
updates and relevant questions will be pushed to experts in the field, and evaluated through
interviews. Everything will be shared with experts in the field, and worked on during at least
one so-called VOCamp (Vocabulary Camp)7. During these events, vocabularies and
ontologies are created and improved in interactive development sessions with a small number
of experts.

The fit and relevance of the model is analyzed in two case studies.

• First, through application in the online peer-based learning community “Peer 2 Peer
University”, the applicability is tested. This case study will involve interviews with the

4
http://p2pu.org/
5
http://www.foaf-project.org/
6
http://wiki.myopen.org/
7
http://vocamp.org/

52
 main developers of the platform, and will evaluate the usefulness of individual elements
and relations described in the model.
• Secondly, applying it in the social academic environment PLoS One8, in Article Level
Metrics may be used to ‘prove’ expertise and a contextualized notion of value of a paper
and its author(s).

3.5.2 Second cycle: developing the reputation system 
The second cycle is aimed at developing the reputation system. The high ambition level of the
research requires us to focus on the most relevant topics. This cycle will mainly be focused on
developing and improving the algorithm that is able to analyze, interpret, and contextualize
reputation statements according to the descriptions and relations in the semantic model. There
are several requirements for this system. As we described in chapter 2.3, the transitivity of
trust (and thus reputation) depends heavily on context. The semantic model of reputation does
not only include the typical value statements, but also describes interoperability with
community-specific ontologies on reputation (statements). The developed reputation system
must therefore be able to combine the context of a reputation statement with the value
expressed. Another requirement is the inclusion of weight into reputation statements.

It is not in the focus of the research to make a user-friendly interface, or reputation

management program. Rather than the technical implementation of the algorithm, we look at
societal implementation and implications, and describe requirements that relate with privacy
and identity provision.

Problem diagnosis 
A system that is able to derive a notion of value or expertise from interactions and activities
by actors in social online environment is very useful in learning environments and
organizations where knowledge transfer and localization is fundamental to the business. In the
problem diagnosis, we will focus on identifying elements in an online reputation that will
provide the highest value for both single users as well as whole communities. This includes
the following.

• An extensive literature analysis on trust and reputation will result in an overview of

existing trust and reputation algorithms, with their merits and deficits. We will match the
requirements described in the semantic model of the previous cycle to determine the
various approaches and develop a new approach.
• In-depth interviews amongst human resource managers and potential employers on
StackOverflow9 will shed light on the requirements on the reputation system regarding
trust in such a system.
• Interviews with members of the Peer 2 Peer University will provide us with insight in user
requirements concerned with privacy and personal information as well as online
representation and ownership of information.

8
http://www.plosone.org/
9
http://jobs.stackoverflow.com/

53
Regarding theory building, this part of the research will advance knowledge on trust and
reputation system by (i) proposing a novel approach for integrating context into a value
statement; and (ii) explicating user requirements on trust and reputation that carries
information about expertise and knowledge.

Technology invention / design  
Using the outcomes of the case study analysis and interviews and literature review on trust
and reputation systems, an algorithm will be constructed that is able to model reputation
based on knowledge-sharing and learning support activities for any p2p learning context. It
includes a way to make the reputation transitive and increase external validation and
usefulness outside the boundaries of a single community.

Using ProLog10, a popular logic programming language used for natural language processing,
theorem proving, expert systems, etc., the algorithm is translated into a reputation system.
Based on the user needs, an algorithm for reputation will be constructed and deployed in the
Peer 2 Peer University. The reputation system accommodates for different contexts with an
ability to use specialized ontologies, improve external validity and trust in the system.

Theory building is focused on showing the relevance of this approach in comparison with
other existing approaches. Also, a method is developed that makes it easier to implement the
algorithm in an existing community.

Technology evaluation 
The final implementation and evaluation of the algorithm happens through different
experiments on Peer 2 Peer University and other peer-based learning places online.

• First, using available data from PLoS One, StackOverflow, and Peer 2 Peer University,
different deployments of the algorithm can be evaluated. Does the algorithm improve
existing user profiles by providing insight in value added to the community? This means
that user profiles are compared, and by means of interviews, evaluated with the end-users.
• Secondly, a course on Peer 2 Peer University, dealing with this subject, is started. In line
with the objective of the research, learning and teaching happen (at least partially) peer-
based. In order to test the resilience of the system, some of the participants will be asked
to game the system (with the objective to improve their own reputation). Others will keep
an eye on irregular behavior, just as what would happen normally.
• Thirdly, by means of a Delphi study, the scalability of the system is tested. Scaling can be
interpreted in terms of number of users, but also the numbers of communities being able
to adopt and deploy the system.
• The fourth evaluation criteria is concerned with the external validity of the reputation
profiles: are they in fact useful for potential employers or other people outside of the
community in which the profile is created? Through a questionnaire amongst reputed
agents as well as potential employers, requirements and circumstances for trust in
reputation profiles will emerge. This can be used for further research and development.

10
http://en.wikipedia.org/wiki/Prolog

54
• Finally, a comparative analysis will be done using Social Network Analysis. The activities
and interactions measured on the Peer 2 Peer University at the beginning of the research
will be compared with interactions at the end of the research.

3.5.3 Research instruments 
Thus, through our research steps, the following instruments are used: case study research,
social network analysis, interviews, expert survey, and literature review.

Case study research 
The essence of a case study, the central tendency among all types of case study, is that it tries
to illuminate a decision or set of decisions: why they were taken, how they were
implemented, and with what result. Thus, a case study is an empirical inquiry that investigates
a contemporary phenomenon within its real-life context, especially when the boundaries
between phenomenon and context are not clearly evident [193]. Case studies will be used for
evaluating and redesigning both the semantic model and the algorithm/reputation system. For
the case studies, we will use the action research approach, in which the researcher is an
integral part of the phenomenon and whose input influences the outcomes [194]. A case study
will be designed, executed and reported on for descriptive purposes (finding data and
instances in which to evaluate and validate our design artifacts). A report will be written about
the case study findings.

With case studies, we run the risk of difficulties with generalizing or different interpretations;
some arguments can be used to justify actual generalization of a concept or requirement, but
mainly it is a choice one makes in the path from generalization (and less insight) to
specialization (and more insight). A major challenge will be to find an adequate problem
situation or inquiry setting. Many limitations, with regards to availability, credibility, location,
suitability, openness and confidentiality affect not only the actual getting of a case but also the
possibility of successfully gaining insight from it.

The case studies in our research will be used to test the design artifacts. The goal of the case
studies is to see which parts of the design artifacts work and don't work, with the purpose of
learning from mistakes, and adapting the artifacts accordingly. Relevant case studies are those
that share relevant data, or offer some way of support within the context of the research. An
essential element must be that reputation could be used as a way to sustain activities and
learning. Ideally, the case studies come from corporations and governmental organizations, as
this will provide us with cases that are both rooted in reality and practicality, and have an
actual need behind them.

The following cases are proposed, but along the research, other cases may be used.

Case 1: Peer 2 Peer University (P2PU.org) is an online community of open study groups for
short university-level courses. The vision of the founders is similar to what is proposed in this
research. It is also a research project, and the founders have expressed their need for the
system proposed in this research.

55
Case 2: StackOverflow is a question answering community, where problems are solved in a
highly decentralized way. The reason for including this initiative is the availability of data
combined with the highly sophisticated reputation system. Reputation in this system is a very
important incentive, and reputation profiles are used as additions to online curricula on the
same site (they also provide a framework for jobs).

PLoS One: The Public Library of Science an international, peer-reviewed, open-access, online
publication. PLoS ONE welcomes reports on primary research from any scientific discipline.
Especially relevant for this research is their 2009 initiative, called “Article-level Metrics”.
This initiative provides "article-level metrics" on every article in their database to help users
determine the value of that article to them and to the scientific community in general.
Importantly, they provide additional and regularly updated context to the article, which
currently includes data on citations, online usage, social bookmarks, comments, notes, blog
posts about the article, and ratings of the article.

Social network analysis 
The technique of Social Network Analysis (SNA) is able to assist in describing and
understanding the patterns of participant interaction in Networked Learning environments
[195]. SNA is a research methodology that seeks to identify underlying patterns of social
relations based on the way actors are connected with each other. De Laat et al (2007) propose
that interactions among participants in Networked Learning communities may be relatively
easily mapped out and explored using SNA, and provide additional useful analytical data
about the activity and relationships of members in online learning communities [195]. SNA
may help in asking further questions about the nature of learning in online communities,
because it provides a new way of viewing participants’ activities. It may also help to confirm
or contextualize conclusions and interpretations about participants’ behavior in these
environments gathered using existing analytical techniques.

Using SNA, the social environment of participants in the Peer 2 Peer University can be
mapped as well as the patterns of relationships among interacting members. This information
can be used to validate the initial conceptual mapping done based on literature research.
Especially interesting will be the intensity of the contact between different members, and the
roles individual members fulfill. Moreover, an initial analysis focused on user participation
may be used at the end of the research to determine the influence of the created artifact on
motivation.

Interviews 
Through interviews with selected semantic web developers and scientists (first cycle), and
potential employers and computer scientists (second cycle), will we be able to evaluate
feasibility and quality of technical solutions, and get insight into trust parameters that need to
be addressed in the solution.

The strength of an interview is that it describes real world situations [196]. The weakness of
this instrument lies in the possible bias of respondents. Interviews will be conducted in a
semi-structured format. Interviews will be tape-recorded, and transcripts will be made.

56
Transcripts will be sent to the interviewees for approval. Interviewees will be selected based
on perceived experience in the field of educational game design, and their history of relevant
publications.

Expert survey 
A first instrument that will be used for an expert survey, is the questionnaire with closed
questions that make use of a Likert scale. A larger group of experts will be questioned;
colleagues, national and international experts, from the corporate, governmental and academic
sectors. Results will be summarized and analyzed through statistical analysis. A second
instrument that will be used for an expert survey, is the Delphi Method; the systematic,
interactive forecasting method which relies on a panel of independent experts [197]. A three-
round Delphi study will be employed to evaluate the semantic model, algorithm, and societal
implications. A report will be written about the results.

Literature review 
Several literature reviews will be conducted; one with a focus on peer-based learning in
online communities, one on semantic annotation of concepts relating with reputation, quality,
and user interest, and one on existing trust and reputation systems.

For categorizing the reviewed literature, a framework will be used to define the (a) goal, (b)
coverage, (c) organization, and (d) audience for the study. The results are organized
conceptually and the relevant works are presented, in accordance to the framework used by
Cooper [198]. Literature will be drawn from journals, conference papers and books
(dissertations, research reports, etcetera).

3.6 Scope of the research  
This research will come up with a novel approach to reputation profiling to be used in peer-
based online communities. The developed semantic model and system will not be a finished
product, and most likely will never be. Rather, if the potential of the research is clear to a
sufficiently large number of practitioners and researchers, then further research and
development is likely to happen. This research will mainly focus on aspects not dealt
(satisfactorily) in this research. Questions that are addressed in the research and will be not
extensively researched include: If a person has different reputations on different, but similar
communities, how can this person aggregate and merge these profiles? If someone joins a
community, how can the existing reputation information, acquired in another community, be
integrated? How does ‘knowledge’ or expertise degrade over time? How should reputation
profiles be dealt with, who will manage the information, and how will this happen? What are
the best interfaces to improve usage of the system?

3.7 Transferability, acceptance and dissemination of research  
In order to ensure continued development on reputation in peer-based learning environment,
several measures will be taken that will improve the transferability, acceptance, and
dissemination.

57
• Open source. This research is inspired by open source economics and organization.
Therefore, it would be contrary to belief when research results or created artifacts are not
shared using no or an open source license.
• Learning community. If the idea catches on, the learning community that emerges will
both develop and use the reputation system.
• Open standards. Following from the above, open standards, and especially standards for
semantic annotation and concept mapping, are used to make the semantic model.
• Master students. The intention is to involve Master students working on their thesis in the
all stages of the research, especially when conducting the case study analyses.
• VOCamps and conferences will be visited and research papers published to address the
Semantic Web community as well as the educational technology community.

3.8 Planning 
The research, as explained above, contains two cycles, aimed at developing a semantic model
(first cycle), and the reputation algorithm (second cycle). Even though the cycles may seem
rather distinct, new insights can affect both artifacts, and therefore the cycles cannot be seen
separately. The planning below uses the steps and results defined in these cycles.

The outline below contains scheduled research activities and their expected results for the
four-year project.

Time Activity Expected results

jul-2009 General literature review Literature review on web-based learning, self-
directed learning
aug-2009
sep-2009
okt-2009
nov-2009
dec-2009
jan-2010 Draft research proposal
feb-2010
mrt-2010
apr-2010
mei-2010 VOCamp Paris Development of conceptual model for User
Interest
jun-2010 Finished research proposal First research proposal, as well as first chapter
of dissertation
jul-2010 Literature review on p2p learning First draft of conceptual model focused on
motivation and typical activities
aug-2010
sep-2010 Social network analysis on P2PU.org Second draft of conceptual model adding
relevance to the typical activities
Overview of typical interactions, and intensity
of interactions, without any reputation system
okt-2010 Baby born A boy or a girl

58
nov-2010 Literature review on semantics for p2p Overview of models and existing ontologies for
learning expertise, trust, and more to learn from/reuse
dec-2010
jan-2011 Case study P2PU.org (interviews) This case study will involve interviews with
the main developers of the platform, and will
evaluate the usefulness of individual elements
and relations described in the model.
feb-2011 Case study PLoS One Article Level Metrics may be used to improve
definitions and relations defined in the model
mrt-2011 Analyzing results Design artifact:Development of the semantic
model for reputation in p2p learning
environments
apr-2011
mei-2011 Literature review on online trust and An overview of existing trust and reputation
reputation systems algorithms, with their merits and deficits. The
existing systems will be matched with the
requirements described in the semantic model
to determine the value of various approaches
and develop a new approach.
jun-2011 Interviews with HRM managers on This will shed light on the requirements on the
jobs.stackoverflow.com reputation system regarding trust in such a
system.
jul-2011 Interviews with members of P2PU.org Provides insight in user requirements
concerned with privacy and personal
information as well as online representation
and ownership of information.
aug-2011
sep-2011 Analyzing results Artifact: initial algorithm
Artifact: theory on online reputation in the
knowledge economy
Artifact: improved semantic model
okt-2011
nov-2011 Development of reputation system Artifact: reputation system (that includes the
algorithm and the semantic model)
dec-2011
jan-2012 Deployment of the system in P2PU.org Improved reputation system (testing algorithm
and ability to game the system)
feb-2012 Deployment of the system in Improved reputation system (applicability in
StackOverflow another context)
mrt-2012 Deployment of the system in PLoS Improved reputation system (applicability in
One another context)
apr-2012 Conducting Delphi study Improved reputation system (scalability of the
system)
mei-2012 Interviews with HRM managers on Improved reputation system (external validity
jobs.stackoverflow.com and trust in the system)
jun-2012 Interviews with members of P2PU.org Improved reputation system (internal validity
and trust by reputed agents)
jul-2012 Social network analysis on P2PU.org Overview of typical interactions, and intensity
of interactions, with the reputation system

59
aug-2012
sep-2012 Aggregating results and writing Initial draft of dissertation
dissertation
okt-2012
nov-2012
dec-2012
jan-2013 Finalizing dissertation Final, revised document
feb-2013
mrt-2013
apr-2013
mei-2013
jun-2013
jul-2013 PhD defense

60
List of references

[1] S. Downes, "Places to Go : Connectivism & Connective Knowledge," Innovate, vol.

5, 2008, p. 6.
[2] J. Bransford, A. Brown, and R. Cocking, How people learn: Brain, mind, experience,
and school, Washington DC: National Academies Press, 1999.
[3] J. Brown and R. Adler, "Minds on fire: Open education, the long tail, and learning
2.0," Educause Review, vol. 43, 2008, p. 16.
[4] G. Siemens, "Connectivism - A learning theory for the digital age," International
Journal of Instructional Technology and Distance Learning, vol. 2, 2005, p. 3–10.
[5] D. Wiley and E. Edwards, "Online self-organizing social systems: The decentralized
future of online learning," Quarterly Review of Distance Education, vol. 3, 2002, p.
33.
[6] A. Koohang and K. Harman, "Advancing Sustainability of Open Educational
Resources," Issues in Informing Science and Information Technology, vol. 4, 2007.
[7] S. Downes, "Models for sustainable open educational resources," Interdisciplinary
Journal of Knowledge and Learning Objects, vol. 3, 2007, p. 29–44.
[8] D. Wiley, "On the sustainability of open educational resource initiatives in higher
education," vol. 26, 2006, p. 21.
[9] I. Tuomi, "Open Educational Resources: What they are and why do they matter,"
October, available at: www. oecd. org/edu/oer, 2006, p. 44.
[10] J. Hylén, "Open educational resources: Opportunities and challenges," Open
Education, 2006, p. 11.
[11] R.G. Baraniuk, "Challenges and Opportunities for the Open Education Movement: A
Connexions Case Study," Opening up education - The collective advancement of
education through open technology, open content, and open knowledge, T. Iiyoshi and
M.V. Kumar, Cambridge, MA: MIT Press, 2008, pp. 229-247.
[12] J. Hylen, Giving Knowledge for Free: The Emergence of Open Educational
Resources, Paris: 2007.
[13] S. Gurell, Open Educational Resources Handbook for Educators, 2008.
[14] M. Mcalpine, "E-portfolios and Digital Identity: some issues for discussion," E-
Learning, vol. 2, 2005, p. 378.
[15] P. van Der Zanden, The Facilitating University, Delft: Eburon Uitgeverij BV, 2009.
[16] J. Schmidt, C. Geith, S. Håklev, and J. Thierstein, "Peer-To-Peer Recognition of
Learning in Open Education," The International Review of Research in Open and
Distance Learning, vol. 10, 2009.
[17] S. Downes, The future of online learning, 2008.
[18] C. Davidson and D. Goldberg, "The future of learning institutions in a digital age,"
The MIT Press, 2009, p. 81.
[19] J.P. Schmidt and M. Surman, Open sourcing education: Learning and wisdom from
the iSummit 2007, 2007.
[20] S. Downes, "The future of online learning," Online Journal of Distance Learning
Administration, vol. 1, 1998.
[21] S. Downes, "E-learning 2.0," eLearn, vol. 2005, 2005, p. 1.
[22] J. Kenway, E. Bullen, and S. Robb, "The Knowledge Economy, the Techno-preneur
and the Problematic Future of the University," Policy Futures in Education, vol. 2,
2004, p. 330.
[23] W. Snyder and E. Wenger, "Communities of Practice," Lessons learned from Auburn
Hills. …, 1999.

61
[24] E. Wenger, R. McDermott, and W. Snyder, Cultivating communities of practice: A
guide to managing knowledge, Boston, MA: HBS Press, 2002.
[25] A. Ardichvili, "Learning and Knowledge Sharing in Virtual Communities of Practice:
Motivators, Barriers, and Enablers," Advances in Developing Human Resources, vol.
10, 2008, pp. 541-554.
[26] M. Soekijad, M.a. Huis in 't Veld, and B. Enserink, "Learning and knowledge
processes in inter-organizational communities of practice," Knowledge and Process
Management, vol. 11, 2004, pp. 3-12.
[27] C. Johnson, "A survey of current research on online communities of practice," The
Internet and Higher Education, vol. 4, 2001, pp. 45-60.
[28] J. Storck, "Communities of practice and organizational performance," International
Business, 2001, pp. 831-841.
[29] E. Wenger, "Communities of practice and social learning systems," Organization, vol.
7, 2000, pp. 225-246.
[30] B. Allan and D. Lewis, "The impact of membership of a virtual learning community
on individual learning careers and professional identity," British Journal of
Educational Technology, vol. 37, 2006, pp. 841-852.
[31] M.J. Hannafin, K.M. Hannafin, S.M. Land, and K. Oliver, "Grounded practice and the
design of constructivist learning environments," Educational Technology Research
and Development, vol. 45, 1997, pp. 101-117.
[32] G. Lefoe, "Creating constructivist learning environments on the web: the challenge in
higher education," Development, 1997, pp. 453-464.
[33] D.H. Jonassen, "Objectivism versus constructivism: Do we need a new philosophical
paradigm?," Educational Technology Research and Development, vol. 39, 1991, pp. 5-
14.
[34] J.R. Savery and T.M. Duffy, "Problem Based Learning: An instructional model and its
constructivist framework," Constructivist learning environments: case studies in
instructional design, B.G. Wilson, 2001, pp. 135-151.
[35] S.A. Barab and T. Duffy, "From Practice Fields to Communities of Practice,"
Learning, 2000.
[36] M. Zarb, "Modelling participation in virtual communities-of-practice," 2006.
[37] A.E. Veldhuis-Diermanse, H.J. Biemans, M. Mulder, and H. Mahdizadeh, "Analysing
Learning Processes and Quality of Knowledge Construction in Networked Learning,"
The Journal of Agricultural Education and Extension, vol. 12, 2006, pp. 41-57.
[38] M.D. Laat, "Networked Learning," 2006, pp. 7-9.
[39] M.D. Laat and V.I. Lally, "Complexity, theory and praxis: Researching collaborative
learning and tutoring processes in a networked learning community," 2003, pp. 7-39.
[40] P.D. Vries, "Networked Learning for Lifelong Learners," Journal of documentation,
2008, p. 26.
[41] I. Tuomi, "Learning in the Age of Networked Intelligence," European Journal of
Education, vol. 42, 2007, pp. 235-254.
[42] B. Zimmerman, "Attaining self-regulation: A social cognitive perspective," Handbook
of self-regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, 2000, pp. 13-40.
[43] D. Schunk and B. Zimmerman, "Self-regulation and learning," Handbook of
Psychology, Volume 7, Educational Psychology, W.M. Reynolds, G.E. Miller, and I.B.
Weiner, Wiley, 2003, pp. 59-78.
[44] W. Veen, J.V. Staalduinen, and T.A. Hennis, "Informal self-regulated learning in
corporate organizations (submitted)," Fostering Self-regulated learning through ICTs
(forthcoming), G. Dettori and D. Persico, Genova, Italy: Institute for Educational
Technologies Italy's National Research Council, 2010.

62
[45] B. Zimmerman, "Becoming a self-regulated learner: An overview," Theory into
practice, 2002.
[46] J. Randi, "Teachers as self-regulated learners," The Teachers College Record, 2004.
[47] J. Meece, "The role of motivation in self-regulated learning," Self-regulation of
learning and performance: Issues …, 1994.
[48] S. Paris and P. Winograd, "The role of self-regulated learning in contextual teaching:
Principles and practices for teacher preparation," Preparing Teachers to Use
Contextual Teaching and Learning Strategies to Improve Student Success in and
beyond School, 2003.
[49] L. Kremer-Hayon and H. Tillema, "Self-regulated learning in the context of teacher
education," Teaching and Teacher Education, 1999.
[50] G. Hardin, "The Tragedy of the Commons," Journal of Natural Resources Policy
Research, vol. 1, 2009, pp. 243-253.
[51] E. Raymond, "The cathedral and the bazaar," Knowledge, Technology & Policy, vol.
12, 1999, pp. 23-49.
[52] P. Himanen, The hacker ethic and the spirit of the information age, Random House
Inc. New York, NY, USA, 2001.
[53] E.V. Hippel and G.V. Krogh, "Open Source Software and the "Private-Collective"
Innovation Model: Issues for Organization Science," Organization Science, vol. 14,
2003, pp. 209-223.
[54] Y. Benkler, The wealth of networks, New Haven, CT: Yale University Press, 2006.
[55] M. Mauss, The gift; Forms and Functions of Exchange in Archaic Societies, New
York: The Norton Library, 1967.
[56] M. McLure-Wasko and S. Faraj, "Why Should I Share? Examining Social Capital and
Knowledge Contribution in Electronic Networks of Practice," MIS Quarterly, vol. 29,
2005.
[57] P. Kollock, "The economies of online cooperation: Gifts and public goods in
cyberspace," Communities in cyberspace, M.A. Smith and P. Kollock, New York:
Routledge, 1997, pp. 220-239.
[58] T.A. Hennis and G.L. Kolfschoten, "Understanding Social Mechanisms in Online
Communities (submitted)," Group Decision and Negotiation 2010, Delft: 2010.
[59] J. Lalley and R. Miller, "The Learning Pyramid: Does It Point Teachers in the Right
Direction?.," Education, vol. 128, 2007, p. 16.
[60] J.S. Brown, "The Social Life of Learning: How can Continuing Education be
Reconfigured in the Future?," Continuing Higher Education Review, vol. 66, 2002, pp.
50-69.
[61] Y. Wang and J. Vassileva, "Trust and reputation model in peer-to-peer networks,"
Proceedings of the 3rd International Conference …, 2003.
[62] T.L. Wentling, C. Waight, J. Gallaher, J.L. Fleur, C. Wang, A. Kanfer, and L.S.
Group, "e-learning - A Review of Literature," 2000, pp. 1-73.
[63] L. Vu, T.G. Papaioannou, and K. Aberer, "Synergies of Different Reputation Systems:
Challenges and Opportunities," 2009 World Congress on Privacy, Security, Trust and
the Management of e-Business, Saint John, Canada: IEEE Computer Society, 2009, pp.
218-226.
[64] M. McLure-Wasko and S. Faraj, "Why should I share? Examining social capital and
knowledge contribution in electronic networks of practice," MIS Quarterly, vol. 29,
2005, pp. 35-57.
[65] P. Kollock, "The Economies of Online Cooperation: Gifts and Public Goods in
Cyberspace," Communities in cyberspace, M. Smith and P. Kollock, 1999.

63
[66] J. Lampel and A. Bhalla, "The Role of Status Seeking in Online Communities: Giving
the Gift of Experience," Journal of Computer-Mediated Communication, vol. 12,
2007, p. article 5.
[67] A. Berlanga, E. Rusman, M. Bitter-Rijpkema, and P. Sloep, "Guidelines to Foster
Interaction in Online Communities," Learning Network Services for Professional
Development, R. Koper, Berlin, Heidelberg: Springer, 2009, pp. 27-42.
[68] S. Downes, "The future of online learning - 10 years later," Half an Hour (Blog), vol.
1, 2008.
[69] I. Tuomi, "Learning in the Age of Networked Intelligence," European Journal of
Education, vol. 42, 2007, pp. 235-254.
[70] J. Preece and D. Maloney-Krichmar, "Online communities: focusing on sociability
and usability," Handbook of human-computer interaction, J. Jacko and A. Sears,
Mahwah, New Jersey: Lawrence Erlbaum Associates Inc. Publishers, 2003, p. 596–
620.
[71] B. Matthews, "Semantic Web Technologies," 2005, p. 19.
[72] J. Hendler, "The Dark Side of the Semantic Web," IEEE Intelligent Systems, vol. 22,
2007, pp. 2-4.
[73] D. Siegel, Pull: The Power of the Semantic Web to Transform Your Business,
Portfolio Hardcover, 2009.
[74] E. Chang, F. Hussain, and T. Dillon, "Reputation Ontology for Reputation Systems,"
Lecture notes in computer science, 2005.
[75] J. Golbeck and J. Hendler, "Reputation network analysis for email filtering,"
Proceedings of the First Conference on Email and …, 2004.
[76] J. Golbeck and J. Hendler, "Accuracy of Metrics for Inferring Trust and Reputation in
Semantic Web-based Social Networks," Lecture Notes in Computer Science, 2004.
[77] L. Vercouter, S. Casare, and J. Sichman, "An experience on reputation models
interoperability based on a functional ontology," Proceedings of The IJCAI-07, 2007,
pp. 617-622.
[78] S. Casare and J. Sichman, "Towards a functional ontology of reputation," Proceedings
of the fourth international joint …, 2005.
[79] P.J. Windley, K. Tew, and D. Daley, "A Framework for Building Reputation
Systems," WWW2007, 2007.
[80] J. Golbeck, C. Park, and J. Hendler, "Inferring Reputation on the Semantic Web," A
Journal On The Theory Of Ordered Sets And Its Applications, 2004.
[81] M. Grobelnik and D. Mladenic, "Ontology Generation from Social Networks," Social
Networks, 2009.
[82] B. Aleman-meza, U. Bojars, H. Boley, J.G. Breslin, M. Mochol, A. Polleres, and A.V.
Zhdanova, "Combining RDF Vocabularies for Expert Finding," Science, 2005.
[83] United Nations General Assembly, "Universal Declaration of Human Rights,"
Wikipedia, 1948.
[84] G. Stahl, "Meaning and interpretation in collaboration," Designing for Change in
Networked Learning Environments: Proceedings of the International Conference on
Computer Support for Collaborative Learning (CSCL ‘03), B. Wasson, S. Ludvigsen,
and U. Hoppe, Bergen: Kluwer Academic Publishers, 2003, pp. 523-532.
[85] T. Caswell, S. Henson, M. Jensen, and D. Wiley, "Open Educational Resources :
Enabling universal education," International Review of Research in Open and
Distance Learning, vol. 9, 2008, pp. 1-11.
[86] Opening up education - The collective advancement of education through open
technology, open content, and open knowledge, Cambridge, MA: MIT Press, 2008.

64
[87] M. Pascual, G. Geser, A. Gruber, S.R. Forschungsgesellschaft, P. Baumgartner, V.
Naust-, L. Pullich, S. Lamminnen, A. Koivisto, and M. Group, "The concept of open
educational resources as instrument for implementing lifelong learning strategies at
higher and further education institutions," Source, 2006, pp. 1-8.
[88] U. Klotz, "The challenges of the New Economy *," 1999, pp. 1-24.
[89] B. Skinner, "Are theories of learning necessary?.," Psychological review, vol. 57,
1950, p. 193–216.
[90] E. Thorndike, An introduction to the theory of mental and social measurements,
Teacher's College, Columbia University, 1913.
[91] D.a. Norman, "Twelve Issues for Cognitive Science," Cognitive Science, vol. 4, 1980,
pp. 1-32.
[92] A. Newell and H.A. Simon, Human problem solving, Englewood Cliffs, NJ: Prentice-
Hall, 1972.
[93] J.R. Anderson, "Acquisition cognitive skills," Psychological review, vol. 89, 1982, pp.
369-406.
[94] A. de Groot, "Thought and Choice in Chess (rev. translation)," Mouton and Company,
The Hague, 1965.
[95] K. Ericsson and N. Charness, "Expert performance: Its structure and acquisition.,"
American psychologist, vol. 49, 1994, p. 725–747.
[96] B. Cole, "Characterizing On-line Communication: A First Step.," 1996.
[97] J. Lave and E. Wenger, Situated learning: Legitimate peripheral participation,
Cambridge Univ Pr, 1991.
[98] J. Lave, Cognition in practice: Mind, mathematics, and culture in everyday life,
Cambridge Univ Pr, 1988.
[99] B. Rogoff, "Apprenticeship in thinking: Cognitive development in social context,"
1990.
[100] J. Tudge and B. Rogoff, "Peer influences on cognitive development: Piagetian and
Vygotskian perspectives," Lev Vygotsky: Critical assessments, 1999.
[101] L. Vygotsky and M. Cole, Mind in society: The development of higher psychological
processes, 1978.
[102] J. Bruner, "The narrative construction of reality," Critical inquiry, vol. 18, 1991, p. 1–
21.
[103] J. Piaget and M. Cook, The origins of intelligence in children, International
Universities Press New York, 1952.
[104] A. Amory and R. Seagram, "Educational game models: conceptualization and
evaluation," Journal of Higher Education, vol. 17, p. 206–217.
[105] B.V. Hout-Wolters, R. Simons, and S. Volet, "Active Learning: Self-directed learning
and independent work," Springer, R. Simons, Kluwer Academic Publishers, 2000, pp.
21-36.
[106] L. Rieber, "Seriously considering play: Designing interactive learning environments
based on the blending of microworlds, simulations, and games," Educational
technology research and development, vol. 44, 1996, p. 43–58.
[107] W. Bouman, T. Hoogenboom, R. Jansen, M. Schoondorp, B. de Bruin, and A.
Huizing, The realm of sociality: notes on the design of social software, Amsterdam:
2007.
[108] M. Ackerman, J.M. Carroll, G. Demichelis, M. Huysman, B. Wellman, and V. Wulf,
"Communities and Technologies: An Approach to Foster Social Capital?," Practice,
2004, pp. 406-408.

65
[109] H. Allert, "Coherent Social Systems for Learning: An Approach for Contextualized
and Community Centred Metadata," Journal of Interactive Media in Education, vol.
2004, 2004, pp. 1-30.
[110] E. Dale and B. Nyland, "Cone of learning," Educational Media, 1960.
[111] S. Paavola, L. Lipponen, and K. Hakkarainen, "Models of Innovative Knowledge
Communities and Three Metaphors of Learning," Review of Educational Research,
vol. 74, 2004, pp. 557-576.
[112] G. Stahl, T. Koschmann, and D. Suthers, "Computer-supported collaborative learning:
An historical perspective," Cambridge handbook of the learning sciences, S. Sawyer,
Cambridge, UK: Cambridge University Press, 1999.
[113] W. Veen, H. Lukosch, and P. de Vries, "Improving organizational learning through
Networked Learning," GI2008, 2008, pp. 22-31.
[114] C. Jones and C. Steeples, "Perspectives and issues in networked learning," Networked
learning, Springer-Verlag New York, Inc., 2001, p. 14.
[115] G. Siemens, Knowing knowledge, Lulu. com, 2006.
[116] K. Illeris, "Towards a contemporary and comprehensive theory of learning,"
International Journal of Lifelong Education, vol. 22, 2003, p. 396–406.
[117] Y. Benkler, "Common Wisdom: Peer Production of Educational Materials," 2005.
[118] T.A. Hennis, E. Sjoer, and W. Veen, "The future of open courseware; A Case Study,"
Proceedings of World Conference on Educational Multimedia, Hypermedia and
Telecommunications, 2008, pp. 686-696.
[119] G.H. Brundtland, "Our Commons Future," 1987.
[120] K. Hargroves and M. Smith, "The Natural Advantage of Nations," Business
Opportunities, Innovation and Governance in the 21st Century, London:
Earthscan/James&James, 2005, pp. 7-25.
[121] L. Ripamonti, F. De Cindio, and M. Benassi, "Online communities sustainability:
some economic issues," The Journal of Community Informatics, vol. 1, 2005, p. 63–78.
[122] J. Giles, "Internet encyclopaedias go head to head.," Nature, vol. 438, 2005, pp. 900-
1.
[123] M.A. Wirth, "An analysis of international quality management approaches in e-
learning: Different paths, similar pursuits," Handbook on Quality and Standardisation
in E-Learning, U. Ehlers and J.M. Pawlowski, Berlin, Heidelberg: Springer, 2006, pp.
97-108.
[124] J. Milam, R.A. Voorhees, and A. Bedard-Voorhees, "Assessment of online education:
Policies, practices, and recommendations," New Directions for Community Colleges,
2004, pp. 73-85.
[125] R. Voorhees, Measuring What Matters: Competency-Based Learning Models in
Higher Education: new directions for institutional research, Lavoisier, 2001.
[126] S. Downes, Open Source Assessment.
[127] J. Koh, Y. Kim, B. Butler, and G. Bock, "Encouraging participation in virtual
communities," Communications of the ACM, vol. 50, 2007, pp. 68-73.
[128] B.S. Butler, When is a Group Not a Group: An Empirical Examination of Metaphors
for Online Social Structure, Pittsburgh: 1999.
[129] T.A. Hennis, "Platform voor promovendi - Valkuilen op de werkplaats," Intellektueel
Kapitaal, 2009, pp. 40-42.
[130] T.A. Hennis, "Platform voor promovendi - Lessons learned leidt tot achtstappenplan,"
Intellektueel Kapitaal, 2009, pp. 36-40.
[131] F. Ortega, J.M. Gonzalez-Barahona, and G. Robles, "On the inequality of
contributions to Wikipedia," Proceedings of the 41st Annual Hawaii International
Conference on System Sciences, 2008.

66
[132] J. Preece and D. Maloney-Krichmar, "Online communities: Design, theory, and
practice," Journal of Computer-Mediated Communication, vol. 10, 2005.
[133] M.J. Culnan and B. College, "Online Communities: Infrastructure, Relational
Cohesion, and Sustainability," Computerizations Movements and Technology
Diffusion: From Mainframes to Ubiquitous Computing, M.S. Elliot and K.L. Kraemer,
New Jersey: Information Today, 2008, pp. 239-259.
[134] T. Yu, L. Lu, and T. Liu, "Exploring factors that influence knowledge sharing
behavior via weblogs," Computers in Human Behavior, vol. 26, 2010, pp. 32-41.
[135] T.D. Moore and M.A. Serva, "Understanding Member Motivation for Contributing to
Different Types of Virtual Communities: A Proposed Framework," SIGMIS-CPR '07:
Proceedings of the 2007 ACM SIGMIS CPR conference on 2007 computer personnel
doctoral consortium and research conference, New York: ACM, 2007, pp. 153-158.
[136] P. Hendriks and D. Vriens, "Knowledge-based systems and knowledge management:
friends or foes?," Information & Management, vol. 35, 1999, pp. 113-125.
[137] H. Lee and B. Choi, "Knowledge management enablers, processes, and organizational
performance: an Integrative View and Empirical Examination," Journal of
Management Information Systems, vol. 20, 2003, pp. 179-220.
[138] H. Lee, "The effects of socio-technical enablers on knowledge sharing: an exploratory
examination," Journal of Information Science, vol. 34, 2008, pp. 742-754.
[139] H.L. Rheingold, "Virtual Communities and the WELL," GNN Magazine, 1993.
[140] E. Pearson, "Digital gifts: Participation and gift exchange in Livejournal
communities," First Monday, vol. 12, 2007.
[141] M. McLure-Wasko, R. Teigland, and S. Faraj, "The provision of online public goods:
Examining social structure in an electronic network of practice," Decision Support
Systems, vol. 47, 2009, pp. 254-265.
[142] M. Endres, S. Endres, S. Chowdhury, and I. Alam, "Tacit knowledge sharing, self-
efficacy theory, and application to the Open Source Community," Journal of
Knowledge Management, vol. 11, 2007, pp. 92-103.
[143] E.V. Hippel, Democratizing Innovation, Cambridge, MA: MIT Press, 2005.
[144] C. Lattemann and S. Stieglitz, "Framework for Governance in Open Source
Communities," Proceedings of the 38th Annual Hawaii International Conference on
System Sciences, vol. 00, 2005, pp. 192a-192a.
[145] F. Labalme and K. Burton, "Reputation capital and exchange mechanisms," 2002.
[146] Y. Wang, "An overview of online trust: Concepts, elements, and implications,"
Computers in Human Behavior, vol. 21, 2005, pp. 105-125.
[147] M. Kinateder and K. Rothermel, "Architecture and algorithms for a distributed
reputation system," Lecture notes in computer science, 2003, p. 1–16.
[148] D. Gambetta, "Can We Trust Trust?," Trust: Making and Breaking Cooperative
Relations, D. Gambetta, 2000, pp. 213-237.
[149] L. Mui, M. Mohtashemi, and A. Halberstadt, "Notions of reputation in multi-agents
systems," Proceedings of the first international joint conference on Autonomous
agents and multiagent systems part 1 - AAMAS '02, 2002, p. 280.
[150] J. Sabater and C. Sierra, "Review on Computational Trust and Reputation Models,"
Artificial Intelligence Review, vol. 24, 2005, pp. 33-60.
[151] P. Resnick, R. Zeckhauser, E. Friedman, and K. Kuwabara, "Reputation Systems,"
Communications of the ACM, vol. 43, 2000, pp. 45-48.
[152] F. Squazzoni, "Review of Reputation in Artificial Societies: Social Beliefs for Social
Order," 2004.

67
[153] a. Abdul-Rahman and S. Hailes, "Supporting trust in virtual communities,"
Proceedings of the 33rd Annual Hawaii International Conference on System Sciences,
vol. 00, 2000, p. 9.
[154] Y. Tan and W. Thoen, "Towards a generic model of trust in electronic commerce,"
International Journal of Electronic Commerce, vol. 5, 2000, pp. 61-74.
[155] A. Josang, "Trust and Reputation Systems," FOSAD 2006/2007, R. Aldini and R.
Gorrieri, Heidelberg: Springer-Verlag, 2007, pp. 209-245.
[156] S.D. Kamvar, M.T. Schlosser, and H. Garcia-Molina, "The Eigentrust algorithm for
reputation management in P2P networks," Proceedings of the twelfth international
conference on World Wide Web - WWW '03, 2003, p. 640.
[157] N. Gal-Oz, T. Grinshpoun, E. Gudes, and A. Meisels, "CROSS-COMMUNITY
REPUTATION: POLICIES AND ALTERNATIVES," IADIS Int. Conf. on Web Based
Communities (WBC2008), 2008, pp. 197-201.
[158] F.R. Farmer and B. Glass, Building Web Reputation Systems, O'Reilly Media, 2010.
[159] S. Brin and L. Page, "The anatomy of a large-scale hypertextual Web search engine,"
Computer Networks and ISDN Systems, vol. 30, 1998, pp. 107-117.
[160] D. Lewandowski, "Web Searching: A Quality Measurement Perspective," 2007.
[161] P. Resnick, R. Zeckhauser, J. Swanson, and K. Lockwood, "THE VALUE OF
REPUTATION ON EBAY: A CONTROLLED EXPERIMENT," Experimental
Economics, vol. 9, 2006, p. 79–101.
[162] T. Malone and R. Laubacher, "The E-lance economy," Harvard Business Review, vol.
76, 1998, pp. 145-152.
[163] D. Florzak, "Are you ready for the e-lance economy?," TECHNICAL
COMMUNICATION-WASHINGTON-, vol. 49, 2002, p. 162–170.
[164] S. Saha, S. Saint, and D. Christakis, "Impact factor: a valid measure of journal
quality?," JOURNAL-MEDICAL LIBRARY …, 2003.
[165] F. Hecht, B. Hecht, and A. Sandberg, "The Journal “Impact Factor” A Misnamed,
Misleading, Misused Measure," Cancer Genetics and Cytogenetics, 1998.
[166] M. Bordons, M. Fernández, and I. Gómez, "Advantages and limitations in the use of
impact factor measures for the assessment of research performance," Scientometrics,
vol. 53, 2002, pp. 195-206.
[167] J. Bollen, M.A. Rodriguez, and H. Van de Sompel, "Journal status," Scientometrics,
vol. 69, 2006, pp. 669-687.
[168] A. Montenegro-Montero, On PLoS' article-level metrics, 2009.
[169] A. Gami, V. Montori, and N. Wilczynski, "Author self-citation in the diabetes
literature," Canadian Medical …, 2004.
[170] D. Harel and B. Rumpe, "Meaningful Modeling : What's the Semantics of
"Semantics"?," Computer, 2004, pp. 64-72.
[171] C. Neylon and S. Wu, "Article-level metrics and the evolution of scientific impact.,"
PLoS biology, vol. 7, 2009, p. e1000242.
[172] N. Shadbolt, T. Berners-Lee, and W. Hall, "The Semantic Web Revisited," IEEE
Intelligent Systems, vol. 21, 2006, pp. 96-101.
[173] J. Breslin, A. Passant, and S. Decker, The Social Semantic Web, Heidelberg: Springer-
Verlag, 2009.
[174] T. Berners-Lee, J. Hendler, and O. Lassila, "The Semantic Web," Scientific American,
vol. 284, 2001, pp. 34-43.
[175] L. Li and I. Horrocks, "A software framework for matchmaking based on semantic
web technology," Proceedings of the twelfth international conference on World Wide
Web - WWW '03, 2003, p. 331.
[176] C. Shirky, "The Semantic Web, Syllogism, and Worldview."

68
[177] J. Hendler, "Semantics and the Network Effect: A little semantics goes a long way,"
IEEE Intelligent Systems, 2002, pp. 2002-2002.
[178] J. Hendler, "It's just a matter of semantics," Fortune Tech, 2009.
[179] R.D. Neufville, "Thesis definition and preparation," 2002, p. 53.
[180] J. Anderson and L. Rainie, "The Impact of the Internet on Institutions in the Future,"
2010, p. 35.
[181] E. Cuba and Y. Lincoln, "Competing paradigms in qualitative research," Handbook of
qualitative research, N. Denzin and Y. Lincoln, Thousand Oaks, CA: Sage
Publications, Inc, 1994, pp. 105-117.
[182] W. Orlikowski and J. Baroudi, "Studying information technology in organizations:
Research approaches and assumptions," Informations Systems Research, vol. 2, 1991,
pp. 1-28.
[183] A.R. Hevner, S.T. March, J. Park, and S. Ram, "Design Science in Information
Systems Research," MIS Quarterly, vol. 28, 2004, pp. 75-105.
[184] R. Flood, Liberating Systems Theory, New York: Plenum Press, 1990.
[185] R. Flood, "Liberating Systems Theory: Towards Critical Systems Thinking," Human
Relations, vol. 43, 1990, pp. 49-75.
[186] H. Sol, "Simulation in Information Systems Development," 1982.
[187] I.I. Mitroff, F. Betz, L.R. Pondy, and F. Sagasti, "On Managing Science in the
Systems Age: Two Schemas for the Study of Science as a Whole Systems
Phenomenon," Interfaces, vol. 4, 1974, pp. 46-58.
[188] S.T. March and G.F. Smith, "Design and natural science research on information
technology," Decision Support Systems, vol. 15, 1995, pp. 251-266.
[189] B. Kuechler and V. Vaishnavi, "On theory development in design science research:
anatomy of a research project," European Journal of Information Systems, vol. 17,
2008, pp. 489-504.
[190] H.K. Klein and M.D. Myers, "A Set of Principles for Conducting and Evaluating
Interpretive Field Studies in Information Systems h," MIS Quarterly, vol. 23, 1999, pp.
67-93.
[191] C. Chorus, B.V. Wee, and S. Zwart, "TPM-catalogue of Concepts, Theories and
Methods," 2010, p. 278.
[192] J. Venable, "A framework for Design Science research activities," 2006 Information
Resources Management Association International Conference, M. Khosrow-Pour,
Washington DC: Idea Group Publishing, 2006, pp. 184-187.
[193] R. Yin, Case study research: Design and methods, Sage Pubns, 2008.
[194] K. Lewin, Resolving social conflicts: and, Field theory in social science, American
Psychological Association Washington, DC, 1997.
[195] M. Laat, V. Lally, L. Lipponen, and R. Simons, "Investigating patterns of interaction
in networked learning and computer-supported collaborative learning: A role for
Social Network Analysis," International Journal of Computer-Supported
Collaborative Learning, vol. 2, 2007, pp. 87-103.
[196] J. Gubrium and J. Holstein, Handbook of interview research: Context \& method,
Sage Publications, Inc, 2002.
[197] H. Linstone, M. Turoff, and others, The Delphi method: Techniques and applications,
Addison-Wesley Publ., 1975.
[198] H. Cooper, "Organizing knowledge syntheses: A taxonomy of literature reviews,"
Knowledge, Technology & Policy, vol. 1, 1988, p. 104–126.

69