S3.1 EJBRM-volume-15-issue-1 (4) QL (3) QT

Editorial for EJBRM Volume 15 Issue 1
April 2017
Editorial by the Editor: Ann Brown
This issue has four research papers –
• Two of which seek to make effective use of the extra-ordinary source of data for qualitative research
now available on the web (Big Data) by two papers (Jeremy Rose and Christian Lennerholt; D A L
Coldwell).
• One presents the idea of a knowledge café as a tool for qualitative research data collection.
(Shawren Singh).
• One is devoted to a recurring problem in using quantitative methods – that of handlling missing
data. (Jasper N Wulff and Linda Ejlskov)
Both Jeremy Rose and Christian Lennerholt (Low cost text mining as a strategy for qualitative research) and D
A L Coldwell’S (Social Physics crowd sourcing and multicultural research practice in the social sciences: E
pluribus unum?) are excited by the potential of the web for providing large data sets to qualitative
researchers.
Rose and Lennerholt consider the web a low cost source. They present a five stage text mining process making
use of available software to mine the web. They have tested their idea by applying it to an example research
problem and this demonstrates that the process is usable but that there are a significant number of problems
in application. Coldwell is concerned about the challenges of research in social science, when combining work
from many sub disciplines, each with it’s own specialists. He proposes creating multi-disciplinary, cross-cultural
collaborative research groups to work with the large data sets available on the web and he describes a case
study of such a collaborative research process.
Collecting qualitative data, especially reports from interviewees, always raises concerns as to whether the
results capture the full experience of the respondents. Shawren Singh (The Knowledge Café as a research
technique) assesses the potential for obtaining a wider range of insights from attendees at a knowledge café.
His paper gives a practical guide to using this technique.
According to Jasper N Wulff and Linda Ejlskov (Multiple Imputations by Chained Equations in Praxis: Suggested
Guidelines and Overview) missing data is an almost unavoidable problem in quantitative data analysis. Their
paper explains the use of Multiple imputation (MI) in helping researchers deal with this problem, and offers
concrete guidelines on how to implement MI-techniques with a focus on multiple imputation by chained
equations (MICE).
ISSN 1477-7029 1 ©ACPIL
Reference this paper as: Brown A, “Editorial for EJBRM Volume 15 Issue 1” The Electronic Journal of Business Research
Methods Volume 15 Issue 1 2017, (pp1) available online at www.ejbrm.com
Low Cost Text Mining as a Strategy for Qualitative Researchers
Jeremy Rose and Christian Lennerholt

University of Skövde, Sweden
jeremy.rose@his.se
christian.lennerholt@his.se
Abstract: Advances in text mining together with the widespread adoption of the Internet have opened up new possibilities
for qualitative researchers in the information systems and business and management fields. Easy access to large amounts
of textual material through search engines, combined with automated techniques for analysis, promise to simplify the
process of qualitative research. In practice this turns out not to be so easy. We outline a design research approach for
building a five stage process for low tech, low cost text mining, which includes insights from the text mining literature and
an experiment with trend analysis in business intelligence. We summarise the prototype process, and discuss the many
difficulties that currently stand in the way of high quality research by this route. Despite the difficulties, the combination of
low cost text mining with qualitative research is a promising methodological avenue, and we specify some future paths for
this area of study.
Keywords: big data, business intelligence, qualitative research method, social media analysis, text mining, text analytics
1. Introduction
Of the two representation systems that characterize research in the social sciences - words and numbers -
qualitative research is the form that relies primarily on the collection, analysis and interpretation of words.
Collections of words texts have traditionally played an important role for the qualitative researcher.
However the developed world has moved on: texts are now predominantly digital, the largest repository for
them is the Internet, and algorithms embedded in software routinely perform the work of text analysis. In this
article we examine the potential of automated analytics to contribute to the work of a qualitative researcher.
Using design science as a simple methodological framework, we investigate the current state of text mining (as
an introduction for the less technical researcher), look at the analytical process as understood by text miners
and undertake our own experiment in trend analysis for business intelligence. The aim of the paper is to
develop a prototype process for qualitative researchers who wish to experiment with text mining as a vehicle
for making parts of their research process more cost effective. We articulate this aim with two research
questions:
RQ1 - how can a low cost, low-tech Internet text retrieval and analysis process be conducted to facilitate
qualitative research?
RQ2 - what are the advantages and disadvantages of such an approach when compared to traditional
ways of collecting and analysing qualitative data?
It will be important that the researcher can still use their hard-earned sense-making skills, does not have to
replace their existing skill set with an entirely new one, and that costs associated with collecting and analysing
data are reduced, not simply replaced by the costs of programming, software and reconstructing texts. These
advances will be important for qualitative researchers in the era of big data analytics as they compete for
legitimacy and attention (not to mention funding) with quantitative researchers skilled in downloading and
processing very large volumes of unstructured data. It will also be interesting for more technically oriented
text miners as they seek to understand the complex overlay of assumptions and interpretations that surround
automated analytics.
We use the simple design science steps proposed by Vaishnavi and Kuechler (2015) to describe the problem
area in more detail, and then to suggest, develop and test a prototype low tech, low cost text mining process.
The paper is organised as follows. Section 2 outlines the design science research approach and the following
sections follow the shape of this approach. Section 3 describes the problem, in terms of potential advantages
of text mining in making
suggestion for the process; section 5 develops the suggestion by investigating the relevant literature, section 6
evaluates the process through a (rather unsuccessful) experiment in using it to examine business intelligence
trends in blogs; section 7 presents the process together with the principle learning from the literature and the
ISSN 1477-7029 2 ©ACPIL
Reference this paper as: Rose J and Lennerholt C, Low Cost Text Mining as a Strategy for Qualitative Researchers The
Electronic Journal of Business Research Methods Volume 15 Issue 1 2017, (pp2-16) available online at www.ejbrm.com
experiments. The conclusion addresses the answers to the research questions and presents future avenues for
methodological research.
2. A design science approach to process building

A process (in this case a process for low cost text mining as a part of qualitative research) can be understood
as a designed artefact for which design science is a suitable research approach. Design science addresses
(Hevner et
al. (2004)). An important strand of the literature is directly concerned with how to enact design research; a
normative literature often expressed as principles or process models (Rose et al., 2010). Vaishnavi and
Kuechler (2015) propose a simple process model involving five steps, between which iteration is encouraged.
1. Awareness of Problem - the identification of a problem that needs a (better) solution. The
awareness of the problem results in an initial solution proposal.
2. Suggestion - a suggestion to how the initial solution proposal should be realized. The suggestion
results in a tentative design.
3. Development - the realization of the tentative design, resulting in an artefact.
4. Evaluation - an evaluation of whether or not the initial problem has been solved by the developed
artefact. The evaluation of the artefact results in some sort of feedback or performance measures
for assessing it.
5. Result/conclusion - the conclusion should be based on the evaluation and should explain the quality
of the solution an assessment of how successful the design research process has been.
Table 1 describes the research approach used for designing a low cost text mining approach suitable for
qualitative researchers.
Table 1: Design research approach to a low cost text mining process suitable for qualitative researchers
Design science step Our research approach
Awareness of Problem Generated from the authors’ extensive experience of qualitative research, and awareness
of rapid advances in the field of text mining
Suggestion An initial process suggestion is developed from text miners’ own accounts of their
process in the literature, adapted for the low cost qualitative researcher context
Development The process suggestion is developed through a small literature study, in which the
research process of selected well-published and relevant text mining articles is analysed
against the suggested process to discover strengths and pitfalls
Evaluation Evaluation is through a case or discovery experiment (Alberts and Hayes, 2002, Alberts
and Hayes, 2005). The method is suitable for early stages of the elaboration of a
research problem, where little is known about conditions, constraints or variables, and
where it is impossible to isolate particular variables for manipulation. Discovery
experiments are designed to encourage creative and innovative problem solving. We
conduct an experiment using the process to analyse trends for business intelligence from
blog data.
Result/conclusion The low cost text mining process is described in terms of the purpose, principle
techniques and tools, outcomes and major problems for each step.
3. Problem awareness: costs of Qualitative research, advances in text mining

Qualitative research employs a variety of research approaches including action research, case study,
ethnography, qualitative surveys, and grounded theory. Researchers collect data through a variety of
techniques including interviews, observation and fieldwork. Written data sources can include documents,
company reports, memos, letters, reports, email messages, faxes, and newspaper articles. These empirical
materials can include visual material such as pictures and videos, but usually the emphasis is on collections of
words and written descriptions - texts. Texts are analysed using a bewildering variety of strategies (analogous
to the many statistical tests available to a quantitative researcher). Miles and Huberman (1994), for instance,
explain more than 40 data analysis techniques. Most forms of textual empirical work display similar
characteristics, including:
 empirical material that is not easily quantifiable

 the search for transferable meanings, such that the material can be summarised and generalised
www.ejbrm.com 3 ISSN 1477-7029

The Electronic Journal of Business Research Methods Volume 15 Issue 1 2017
 representations of (summarised) data within the materials, by means of notes, narratives, graphs,
diagrams or matrices, amongst other techniques
 the analysis of patterns in the data which signify more than individual episodes in the material
 the development of concepts, categories, constructs, variables, or codes which are able to
summarise and generalise the material
 the development of relationships, associations, conditions for the concepts, codes, variables, etc.
 formulation of theory derived from patterns, concepts and relationships
The qualitative researcher is judged by their ability to make sense of volumes of text that would be
overwhelming to an untrained person. However qualitative data are not cheap to acquire and process. Most of
the costs are incurred as researcher time, and they include
 making contact and interacting with research subjects and organisations

 collecting documents, making observations, conducting interviews, taking notes, collecting emails,
examining web-sites
 putting data in textual forms (normally digital) for analysis, transcribing interviews and notes
 buying analysis software of various kinds
 undertaking the analyses, for example coding.
These costs are substantial; we estimate the costs of a moderate size interview survey conducted recently by
one of the authors (Rose et al., 2016) as approximately 1000 hours, or 20 weeks of researcher time.
Researchers have long turned to software to automate parts of these tasks; in our survey part of the analysis
was facilitated by a content analysis software application.
Recent developments in text mining, and the widespread availability of large quantities of empirical material
(textual and non-textual) on the Internet prompt consideration of whether these resources are well-spent.
This may be an old-fashioned way for qualitative researchers to use their time, and it might be more effective
to use the internet as a primary empirical resource and as automate much as possible of the analysis work
with text mining techniques. This strategy eliminates most of the work in our estimate: materials are
downloaded direct from browser to computer in a digital form, large volumes of text and other materials are
easily collectable, no investment in making contacts, interviewing or transcribing, analysis automated or
partially automated with algorithms and software. The potential advantages of an internet/text mining
research strategy for qualitative researchers are considerable.
4. Process suggestion: the internet as a Text source; text mining and analytics
Text mining and analytics covers that part of digital investigation which deals with primarily unstructured or
semi-structured web content, as opposed to structured content in databases, or web structure and usage
mining (Kosala and Blockeel, 2000). Mining develops the analogy of extracting something valued (information,
intelligence) from a larger mass of less valued material (e.g. the world wide web), whereas analytics in this
context is associated with data processing partially or wholly automated with software. Taken together, they
might encompass statistical natural language processing, information extraction, question-answering systems,
opinion mining, sentiment/affect analysis, web stylometric analysis, multilingual analysis, text visualization
(Chen et al., 2012), subjectivity analysis, market sentiment analysis, topic modelling (Pang and Lee, 2008),
dialogue act classification (Kaiser and Bodendorf, 2012), text summarization and a variety of other techniques.
According to He et al. (2013), text mining is largely data driven and its main purpose is to automatically
identify hidden patterns or trends in the data and then create interpretation or models that explain interesting
patterns and trends in the textual data Wikipedia defines text analytics as a set of linguistic, statistical, and
machine learning techniques that model and structure the information content of textual sources for business
intelligence, exploratory data analysis, research, or investigation. Many advances in these techniques are
driven by research in computational linguistics, natural language processing and machine learning. Text mining
and analytics are sometimes associated with big data (Chen et al., 2012, Xiang et al., 2015), as automation
becomes imperative with larger volumes of data. He et al. (2013) ocial media data
are usually large, n -consuming if we had to
www.ejbrm.com 4 ©ACPIL
4.1 T
The existing text mining literature is primarily concerned with developing the algorithmic techniques for
analysis, rather than conducting business and information systems research. However several text-mining
researchers describe their own process. Kosala and Blockeel (2000) propose the following steps:
 resource finding (retrieving relevant documents)

 information selection and pre-processing (removing noise and irrelevance from the text and
presenting it in a form that can be processed by a computer algorithm)
 generalization (discovering patterns)
 analysis (validation and/or interpretation of the patterns)
The first two steps are represented as data capture by Vitolo et al. (2015), to be followed by two steps
concerned with identifying patterns: processing (running software algorithms on the data) and visualisation
(software techniques for making patterns in data more obvious to the researcher). The final outcome of the
process is specific for Xiang et al. (2015): actionable intelligence insights that can lead to decisions or
improvements in the outside world. Their process reads: text pre-processing, processing/analysis, actionable
intelligence. A final contribution is from web content analysis in a more familiar qualitative context (Romano et
al., 2003):
 elicitation (finding and downloading relevant data)

 reduction (analysis activities: selection, coding, clustering the data)
 visualization (as described above)
Though differing in detail, these accounts show some obvious common features, which are adapted for the
process design suggestion in the next section.
4.2 Process suggestion for qualitative research using text mining
T
the data, retrieving it, analysing it and attributing meaning to the analysis. The first step (finding the problem
to address) is implied but never formally stated and the resulting five-step model is adapted to the qualitative
 Problematisation - formulating a research problem (objective or question) that can be solved

(achieved or answered) using some combination of text mining and qualitative data analysis
 Choosing data sources - defining a set of relevant internet text sources with the potential to solve
the research problem
 Retrieving text - downloading, cleaning (removal of irrelevant text, or repair of damaged or difficult
to analyse text), organisation and storage of the set of relevant text sources, such that they can be
analysed
 Analysing text - enacting some (combination of) automated and/or manual analyses on the text
body judged likely to solve the research problem
 Interpreting results - solving the research problem by attributing meaning to the results of the text
analyses.
5. Development: five process steps in the text mining literature

We investigate these process steps in some articles published in quality research outlets, in order to sensitize
ourselves to issues experienced by fellow researchers. We selected articles with extended empirical examples
and a focus on methodological issues. The issues are clarified at the end of each sub-section.
5.1 Problematisation
In this section we look at how text mining researchers characterize their research problems. Xiang et al. (2015)
investigate hotel guest satisfaction, through the medium of travel advice sites facilitating customer reviews
and ratings. Moon et al. (2014) also focus on customer reviews, determining the impact of product reviews on
sales. Attitudes (opinions, sentiment) registered in social media text are a common topic: three pizza chains
are compared though their Twitter and Facebook sites by He et al. (2013), and Kaiser and Bodendorf (2012)

study attitudes towards iPhones in online forums. Apple s support communities for iPods were mined by
Abrahams et al. (2014), who provide a framework for product defect detection. The derivation of demographic
profiling for twitter users is mapped by Ikeda et al. (2013). Several of our examples use article databases as
their sources, for detecting terrorism related texts in news articles (Choi et al., 2014), and for automating
conventional scientific literature studies (Moro et al., 2015, Delen and Crossland, 2008). In many articles the
empirical examples are more demonstrators for a new analytical technique (or combination of techniques)
than serious investigations of external phenomena, and they often display a data-driven (qualitative research:
grounded) approach, working from the available data to derive results, rather than a hypothetico-deductive
approach working from theory to derive hypotheses and testing them. This leads to many very open and
exploratory research questions, for example: What patterns can be found from their Facebook sites
respectively? What patterns can be found from their Twitter sites respectively? (He et al., 2013). It also meant
that problematisation was often poorly related to substantive theory in the problem area.
In summary a large variety of themes and issues can be addressed with text mining, but the exploratory data-
driven character of the research style may make it difficult to engage with the central theoretical issues of a
research field.
5.2 Choosing data sources
Textual data is available from a staggering variety of sources, both internal to organisations (social network
feeds, emails, blogs, online forums, survey responses, corporate documents, news, and call center logs) and
external. In research on developing and testing new algorithmic techniques in the text analytics field the
choice of data source may be relatively insignificant, however this is not the case in research in other fields
where text mining is used as the research method. The choice of data sources becomes important because
most of the analytical techniques used are statistical and/or algorithmic in nature, implying that the
composition of the sample text sources is likely to affect the analysis results (Krippendorff, 2004). The data
sources should ideally be representative and relevant, where an unspoken assumption in many studies is that
the chosen sources represent what happens on the net, and that what happens on the net also represents the
physical world.
In respect to representativeness, Krippendorff (2004) identifies a variety of sampling techniques as appropriate

to text; however none of the text mining articles that we investigated addressed this issue. They made an
unspoken assumption that patterns they found in their Internet data selection would naturally hold true for
larger populations.
If the sampling problem is less critical in the case of big data analyses, where the volume of data sometimes
ensures better representativeness than other research techniques, the relevance problem is more acute.
Human analysts both organize the collection of data so that it is relevant, (with for example a questionnaire)
and intuitively sort and discard irrelevant text during analysis. Automated techniques do not do this except
where they are programmed to do it, so the presence of significant quantities of noise in the data will also
affect the analysis results. A commonly used way of ensuring some degree of relevance was to choose a
themed network site, blog site A (Abrahams et al.,
2014), or iPhone blogs (Kaiser and Bodendorf, 2012). Sometimes a specialised search engine was available,
such as Google scholar for scientific articles (Delen and Crossland, 2008). Basic sampling and relevance
reasoning was sometimes explicit: X (www.xanga.com) as our source of blog data Xanga is
the second most popular blog hosting site after the Google-owned Blogger (www.blogger.com). It is also
ranked 17th in traffic (visit popularity) among all Web sites E (Chau and Xu, 2012). However the
dominating logic for choice of sources in the articles we investigated is convenience, of which a large part
resides in the retrieval and download mechanisms available.
The qualitative researcher will need to address issues of representativeness and relevance.
5.3 Retrieving text
In retrieving text two problems predominate, those of structure and noise. Where some researchers are
prepared to manually download PDF files, most found an API or web crawler an acceptable solution and some
went to considerable lengths to code rule sets and procedures in commercial web mining tools (O'Shea and
Levene, 2011). There is also a legal and ethical context for what may be scraped, and its probable that
copyright is widely abused. Although commonly described as unstructured, textual data on the web often
contains many forms of structure. In addition to the naturally occurring grammatical, lexical and genre
structures of text (scientific articles have titles, authors, abstracts and key words), most web text is annotated
(with html), it contains other significant relationships such as URLs and links, and some is thematically
structured (with xml, for example). Expedia reviews (Xiang et al., 2015) have rating systems, various kinds of
tagging and labelling, including date stamps and author tags, and the free text review typically contains less
text than the tags. The structure can either be retained and exploited (by transposing it to a database for
example), or partially or wholly removed.
Structure that is superfluous to the analysis task (for instance the headers of email messages) is noise, as are
many other textual features that may get downloaded annotations, spacing, links, repetitions, trademarks,
commercial interests promoting products and services, and advertisements. Automation is required for these
tasks, otherwise the costs of retrieving, cleaning, and storing data are considerable for instance in
programming or training a web crawler, removing superfluous annotation characters, checking for download
errors, introducing analysis tags (such as a unique identifier for individual blogs), concatenating files, OCR
processing and other mundane tasks. Structure and noise are related to relevance and representativeness;
unwanted structure and text compromise statistical analysis. Removing them can involve many manual, even
subjective the full set of all discussion threads, as of June 2010 was
crawled and extracted ... in total, 1,560,379 ... Threads containing less than 50 words or less than two postings
were excluded ... we filtered the dataset to identify threads related to defects ... determined the top 200
component description keywords ... ascertained the number of hits for this shortlist sorted the threads from
highest use to lowest ... finally, we re-balanced the dataset by sampling an equal number of threads from
(Abrahams et al., 2014).
The notion that text in large quantities is easily available from the Internet is therefore naïve; researchers must
deal with structure and noise.
5.4 Analysing text
Most of research into text mining goes into the development of algorithms for processing text. Though
academic fields often develop a consensus about how to use research methods (for instance which statistical
tests are appropriate for variance analysis of questionnaire data), text mining is a field in rapid development,
and no such consensus yet exists. All algorithmic techniques rely on assumptions that should condition the
interpretation of their results, most have accuracy limitations, and some rely on intensive computing resources
unavailable to most researchers. A simple analysis technique is word frequency counting, which relies on two
limiting assumptions: that word frequency has some significance in a corpus of texts, and that words have no
relationship to each other the text is treated as a bag of words (unigrams). Bigrams or other n-grams can also
be used. Accuracy is often measured by precision (the fraction of retrieved instances that are relevant), and
recall (the fraction of relevant instances that are retrieved) or their weighted average: F score. The most
common evaluation technique is no compare the performance of the algorithm with results obtained from
human evaluators. In the case of this very simple algorithm (word frequencies), recall and precision can be
expected to be close to 100%. Since many common words have no special significance (the, it) a stop-word list
can be used, and since words often have several variations (like, likes), stemming reduces variants to the basic
stem. Simple calculations (word frequency) can be used in metrics such as tf-idf (term frequency-inverse
document frequency) - intended to reflect how important a word (for instance a search term) is to a corpus of
texts - and incorporated in many search engines.
Text analytics provide varying ways of accomplishing tasks familiar from statistics, machine learning and
natural language processing, many of which also overlap with data mining techniques. Machine learning may
be supervised (dependent on a training set of data pre-coded by a human), semi-supervised or un-supervised.
Common text analysis tasks include classification, clustering, entity recognition, relationship extraction, text
summarization, question answering and sentiment analysis. Classification algorithms assign documents to
predefined categories representing their content (or other parameters such as style, author, language, spam).
Common methods include naïve Bayes, support vector machines, k-nearest neighbour, decision trees,
expectation maximization, latent semantic indexing. Clustering analysis has the advantage of uncovering
unanticipated trends, correlations, or patterns from data (He et al., 2013) using techniques such as decision
tree construction, rule induction, clustering, logic programming, and statistical algorithms. A related task is
topic modelling (Blei, 2012). Information extraction algorithms structure textual data, often through entity

recognition (ER) classifying text stumps into categories, and relation extraction (RE) specifying pre-defined
relationships (Gandomi and Haider, 2015). Text summarization techniques aim to convey key information in
single or multiple documents. Extractive summarization involves determining the most salient text units
through their location and frequency and concatenating them. Abstractive techniques rely on Natural
Language Processing (NLP) to parse the text and derive semantic information (Gandomi and Haider, 2015).
Question answering (QA) systems (such as A “ IBM W provide answers to questions posed
in natural language, making them reliant on NLP techniques, at least for parsing the questions. Modern
systems tend to use NLP techniques to provide semantically derived queries, trawl the web to extract
candidate answers, and rank them to formulate answers. Sentiment analysis (opinion mining) techniques
analyze text for opinions about phenomena such as products and services. Texts can be classified as negative
or positive, or into more complex rating systems. Fundamental techniques are classification, regression and
ranking (Pang and Lee, 2008).
Tasks normally have many algorithmic variations for their solutions - some are minor variations, others have
completely different strategies. Different algorithms (obviously) produce different results. For example, Lu et
al. (2011) compare four related topic modelling strategies: latent Dirichlet allocation (LDA), local LDA, multi-
grain LDA, and segmented (using a Poisson Dirichlet process). Considerable mathematical sophistication is
required to distinguish the different assumptions built into these variations. In addition many algorithmic
techniques require input parameters (the number of neighbours to be considered, which weightings, or how
many topics to be generated) seed words, dictionaries or pre-coded learning materials, requiring a degree of
experimentation from the text miner. These make it difficult to independently assess the accuracy of the
results. Some algorithms, such as LDA, are known to be sensitive to initial conditions. Assessing accuracy is in
any case problematic; where a result coded by humans is available for comparison, accuracy expressed as
precision and recall should reach 70% to be worth publishing, and 80% is considered excellent. Somewhere
between a fifth and a third of results are in these cases erroneous. However in many areas agreement
between humans (inter coder reliability) is known to be poor, making the measurement standard unreliable,
and in many cases human solutions are not available.
One response to accuracy problems is to combine techniques, for example Abrahams et al. (2014) combine the
analysis of lexical, stylistic, social, sentiment, product and semantic features with distinctive terms. Automated
text analysis techniques may be combined with other algorithmic strategies, with more conventional analysis
of structured data, or with traditional qualitative analysis (e.g. coding). Here there is often a trade-off:
accuracy is improved at the cost of transparency in the process, and the overlaps and combination weightings
become difficult to understand, or questionable.
The difficulties facing the naïve researcher in the analysis step are therefore considerable, and can be
summarised in these questions:
 which analysis task is appropriate for the research problem?

 which algorithm(s) should be chosen for this task?
 which starting parameters should be employed (where these are necessary)?
 how reliable are the results?
5.5 Interpreting results
The ability to process a large volume of textual data is only meaningful if the results can contribute to some
business or research activity. Visualization and sense making are key components of interpretation.
Visualization involves organized, compressed assemblies ... that permit conclusion drawing and action´
(Romano et al., 2003) such as extended text, matrices, graphs, and charts. Visualizations organize data into
accessible compact forms in which analysts can identify patterns. Contemporary visualisations include tag
clouds and visual text analysis systems (Dou et al., 2013). Cui et al. (2011) develop a visualisation system for
evolving topics as they develop over time in social networking systems. Despite advanced visualisation
techniques, sense-making problems associated with the results of text analysis are considerable. For instance,
a (typical) topic in the Delen and Crossland (2008) analysis of scientific literature is represented by the
keywords: error, discipline, mis, major, methodology, field, value, time, future, set. A considerable interpretive
leap, on the basis of extensive domain knowledge is required to understand what such a theme might be
about. Many studies lack a domain theory background, which also limits sense making. Since the techniques
are still evolving, studies tend to focus on justifying the usefulness of the text-mining techniques employed,
and the interpretation of the text mining results becomes a secondary consideration. Xiang et al. (2015)
explain the root of many of these difficulties: whereas conventional methods usually start from a set of
predefined hypotheses (derived from existing scientific knowledge, which also set the frame for
interpretation), big data analytics reveal patterns that the researcher must later try to interpret or explain,
sometimes by fitting them to a theoretical context.
Any assumption that text mining will necessarily provide self-explanatory results is also naïve: the text miner,
like any other researcher, must strive to attribute meaning to their results.
6. Evaluation of the text mining process: business intelligence trend analysis experiment
We applied the low cost, low-tech text mining process to a relevant example. The low cost strategy means that
we should not learn new programming skills, buy expensive software, use techniques with high learning curves
or invest more than 15-20 hours of analysis time on a particular text mining technique without productive
sense-making results. In addition we should use only the computing resources available to us on our standard
specification laptops. We allotted roughly one half man-day per week over the course of an academic year. We
chose to conduct trend analysis for business intelligence derived from blogs. Business intelligence
-oriented data as well as the analytical
technologies and computing appro (Davenport and Harris, 2007).
However the experiment was rather frustrating.
Trend scouting aims at discovering topics and opinions which are of current interest and which may evolve in
the future. The web is an important platform for trend scouting since trends in consumer behaviour often arise
in online communities before spreading through society (Kaiser and Bodendorf, 2012). In our text mining
experiment we asked the research question: what current trends in Business Intelligence can be revealed by
mining the Internet? We chose an open question, for which the answer could reasonably be data driven, and
had no formal hypothesis or theoretical background for the study, though we had enough background in the
subject to be able to interpret the results.
We chose, for convenience, major themed blogs: Smart Data Collective, TechnoSocial Blog, Integrate The
Clouds, Key2 Consulting, Data Doghouse, Business Intelligence: Process, People and Products, BI Scorecard,
David Menninger, Meta S. Brown, Peter James Thomas and msSQLgirl. There are several blogs about business
intelligence and we chose these because they are relatively large, with many current and recently updated
posts. The blogs were written in the period 2010-15, with the largest concentration from 2103-4. We have no
economically feasible means of investigating how representative this text sample is, and ignored this issue. We
also ignored copyright and other potential legal issues. Choosing a themed blog ensures some degree of
relevance: however there were many issues with noise deemed likely to interfere with statistical analysis, from
surplus html characters, to blog headers and identifiers, all the way through to commercial interests
repeatedly using the blog as a marketing channel.
6.3 Retrieving text
The blog was scraped using the free crawler tool Import.io. The tool was trained to recognise the structure of
the different sites and retrieve (mainly) the blog entries with some header information. Many of the sites
crawled chose to display only the first 5-6 lines of the blog on their top-level page (with an option to click
through to the rest), and this is all we collected. The results were downloaded as an Excel binary .xsl. file and,
after initial cleaning, transferred to a text file. The downloads had a certain structure (Input, Result, Number,
Widget, Data Origin, Result Row, Source Page URL, data=blog text), which was of no particular analytical value
to us (see Appendix 1). A variety of unpredictable noise was removed manually at this stage, particularly
advertising, page formatting info, U‘L s, links, some company names and authors. There are also many
download and OCR errors too numerous to correct manually. The resulting corpus file contains
approximately 12600 blog entries (example in appendix), 4736 pages, more than three million words.

6.4 Analysing text

We performed a simple frequency analysis of single words by importing the text into a familiar free cloud tool
(Hypertext) after removing stop (common) words. There were many problems with noise, and with clearly
irrelevant words (e.g. company names, blogger names), which we removed iteratively until we had a result
that contained no obviously irrelevant words. We had to make an instinctive guess about how many words
should be in the picture. W ‘ M he
addition of stemming produced similar, but not identical results. We tried to generate n(2)grams to catch
terms such as starter package ran out of memory. We tried a variety of free cloud-based
tools (Voyant, Textalyser, Google spreadsheet text analysis add on), but these could not handle a large volume
of text, or required payment. We also found a simple program (VOSviewer) that generated word clusters,
which we tried to use for topic analysis. Here the algorithmic process was black boxed. The package offered a
variety of settings and alternatives including (for every cluster map) two counting methods, minimum term
occurrences, and number of terms to be selected. The cluster generation algorithm turned out to be sensitive
to these initial conditions, allowing the generation of substantially different visualisations from the same data
set (see next section). CasualConc offered many further analysis possibilities: word concordance, collocation,
clustering, facilitating a nearly limitless investigation; we
Concordance provided more than 6000 contexts for the term in the text corpus. Collocation provided lists of
adjacent words or phrases (such as the bigrams: business analytics, cloud computing, information
management, data warehousing, business performance, industry analysis, predictive analysis, big data). Five
word clusters (frequency > 100) including business intelligence were
 performance tags business analytics

 big data business analytics
 information management it performance
 cloud computing data integration
 business technology chief information
We used some simple word cloud software to visualise the most frequent terms (Figure 1) in our word
frequency analysis.
Figure 1: Word cloud of most frequent single word terms in the BI blog dataset
The word cloud provides some pointers to possible trends in business intelligence but does not, in itself, make
a viable analysis. The word data appears more than 78000 times in the text making some kind of sampling
process necessary for deeper (qualitative) inspection of the text. We experimented with three looking at
random instances in the text, systematic sampling (taking every hundredth appearance) or looking for clusters
of the word in particular blogs. Frequent references to data were neither surprising or (for our purpose)
interesting, so we used our intuition to look at promising word combinations: data integration appeared over
9000 times, and big data nearly 7000 times, making them candidates for trends. Cloud appeared nearly 50,000
times, and although about 7000 of these were in the repeating header of a cloud themed blog site (Integrate
the Clouds) this seemed an excellent trend candidate. Qualitative investigation showed that integration was
nearly always in the context of data integration, system integration or cloud integration (though this last may
have been a statistical distortion caused by the themed blog site mentioned above). Analytics is a term that
causes no great surprise, however about a third of these references were to predictive analytics another
candidate trend. Social appeared more than 5000 times, media more than 3000 we used our intuition to
83
arrive at the term social media another trend candidate. However, with more than 10 available
combinations of the 50 terms in the picture, intuition seemed a poor way of making a comprehensive analysis,
even when supported by the clustering analysis in the next paragraph. The appearance of the term collective
was intriguing, but only the product of poor data cleaning. It appeared three times in the header of every blog
at one of the larger sites Smart Data Collective, accounting for nearly all of its appearances. This also ruled out
the term smart. Other meaningless statistical blips were loading leave and blog.
Figure 2: Two cluster maps of the BI blog data set from VOSviewer showing sensitivity to initial conditions
(different parameter settings)

Our VOSviewer results produced impressive coloured cluster charts, and the unknown algorithm did produce
bigrams 2 (or more) word phrases. Some of these were helpful (data governance, customer relationship
management, business process), some of them still the result of poor data cleaning (smart data collective, next
post), many rather irrelevant to our purpose (open world - O
quite mysterious (KPIs Data Governance Litmus Test). We hoped that statistical clustering would provide some
insight into BI trends but these maps turned out to be exceptionally difficult to interpret. The two considerably
different cluster maps in figure 2 are both produced from the same text corpus by changing some of the
minimum term occurrences, and number of terms to be selected.
We had no way of determining the optimal starting parameters from the very large number of possible
combinations. In figure 2 (top diagram), it may be possible to discern a purple cluster that might be something
to do with the cloud, a blue cluster referring to the business, management and governance side, possibly a
turquoise cluster relating to big data technologies however some groupings are fairly baffling. What should
one make of cluster 11: benioff, big data appliance, cloud offering, configuration, exadata, exalytic, hardware,
keynote, memory capability, noSQL, open world, oracle database, public cloud, ram, social network,
Wednesday? Moreover the analysis program is a black box, so that the statistical process is not available for
inspection, and it is sensitive to initial conditions in this case its settings. It may be possible to produce a
meaningful output by many hours of experimentation, and cleaning or manipulating the data but this is not a
low cost strategy. M methodologically secure in research terms.
We arrived at some candidates for business intelligence trends including: data integration, systems
integration, cloud integration, social media, big data, business management and governance. However, since
we had no secure method for arriving at a convincing result in a reasonable amount of time we abandoned our
investigation. We compared our data analyses with a commercially produced list of BI trends for 2015 from
Tableau Business Intelligence (http://get.tableau.com/campaign/business-intelligence.html) - no research
method reported. We could observe some commonality (governance, social intelligence, analytics, data
integration, cloud analytics) and some differences (journalism, mobility, self-service).
7. Results: a generic process for low cost text mining

In this section we note the strengths and weaknesses of the process we developed, as shown in our evaluation
experiment in trend analysis for business intelligence. Most of the issues we identified in our literature review
in section 5 also became problematic in the experiment.
Though we had no problem finding an open-ended investigation topic for our relatively trivial example, it
etico-deductive style of research prevalent in our research
fields. Here investigations are more normally based on theoretical premises or literature studies, and the
expectation is that research questions are focused, clearly defined and theory based. There may or may not be
an explicit hypothesis, but examples of data-driven statistically generated theory are still rare. Its not
immediately clear how the text mining techniques we studied can be adapted to hypothesis testing, or to
focused research questions, and we found few examples of this in the literature.
The strength of the text mining approach is the easy availability of very large quantities of digital text through
the Internet: however this can be also a weakness in statistical approaches. We chose some thematically
targeted blogs to help improve relevance; nevertheless our blog set included large amounts of irrelevant data.
Nor is it feasible to establish the total population of relevant blogs, so the sampling process is insecure, and
representativeness unknown. Validity often rests on the argument that there is a lot of data. Qualitative
researchers are used to smaller quantities of data collected in some well-targeted way.
7.3 Retrieving text
When compared to interviewing subjects and transcribing the results, retrieving text from the Internet is
relatively quick and easy, though not without costs. Whereas the interviewer has the opportunity to create
their own structure in the resultant texts through the design of questions, it s difficult to know what to do with
the many pre-existing structures of the web. However the predominant problem is noise. An experienced
interviewer will gently guide their research subject back on track if they wander; bloggers are free to discuss
what they choose and site designers add many other noise elements. Since noise undermines statistical
analysis this is not an insignificant problem: efficiency gains in downloading text are reversed if the removal of
noise cannot be cheaply automated.
7.4 Analysing text
In principle the application of text analysis is made insecure by the lack of an agreed, methodologically valid
approach for the field. Instead there are a multitude of competing algorithmic techniques for multiple analytic
purposes, many of them requiring significant researcher input. These remain largely black box techniques for
everyone except algorithm developers and statisticians. In the social sciences there are statistical analysis
techniques for survey data that are generally agreed to be methodologically valid, but no such consensus
exists yet for text analysis. In practice, only rather simple analysis techniques are available at low cost. Where
a variety of analysis techniques were available, the analysis possibilities quickly snowballed out of control. We
seldom knew which analysis task was appropriate, which algorithm pre-programmed software was running,
how we should set starting parameters, or how reliable the results were. Poor execution of the earlier retrieval
stage often led to the junk in, junk out condition. O
without extra cost. A further problem is that there is no standard method for combining statistical text analysis
with more traditional qualitative analysis.
Interpreting text mining results is much more complex than indicated in the literature. Many of our results
required leaps in sense making that even hardened qualitative researchers were unwilling to make. More
experience in problematisation, choosing sources, and retrieving and analysing text would no doubt make
interpretation easier (for example by removing noise from the data). Iterative passes with the data where the
analysis is refined to improve sense-making (for instance by tweaking starting variables for algorithms) may
help. Here the complementary skills of the qualitative analyst may be important. However the hermeneutic
styles of qualitative analysis (for example establishing a relationship between the whole text and its parts)
were seriously challenged by our dataset. It was too large, and too diverse to develop any coherent overall
impression.
Table 1 shows the generic process for low cost text mining for qualitative researchers, comprising the steps:
problematisation, choice of data sets, retrieval of text, text analysis, interpretation of results. For each step,
conclusion about its purpose, available tools and techniques and the principle issues that researchers should
expect to encounter are specified.
Table 1: Generic process for low cost text mining for qualitative researchers
process step purpose tools and techniques outputs principle issues
problematisation focus the area of guess + stepwise research question or finding suitable open
investigation refinement, literature topic questions that can lead
survey, existing market to publishable research
research
choice of data establish a body Google search algorithm text corpus establishing relevance
source(s) of text to be or other techniques for and representativeness
mined prioritising web sources,
sampling, specialised
databases
retrieval of text retrieve, clean crawling, scraping, clean digital text corpus distinguishing
and store the text application programming ready for analysis meaningful and relevant
body interface, database calls, text and structure from
many simple free cloud noise
and desktop tools
available
text analysis model the text word count, topic various (often choosing techniques
modelling, sentiment statistically derived) and algorithms,
analysis, (countless models of the text, text statistical validity, junk in
other techniques), some outputs suitable for junk out
free tools available qualitative analysis
interpretation of learn in order to visualisation techniques, complementary or visualisation, sense
results contribute to further qualitative guiding insights for making
knowledge or analysis, relation of qualitative analysis
guide decisions, model to context or
theory

8. Conclusions
Quantitative text analysis has been around for a long time, but has mainly been the preserve of highly trained
researchers using specialist techniques, and custom software. The Internet has made large quantities of
textual data readily available, and the big data revolution has created the expectation that researchers should
capitalise on new ways of creating knowledge. We set out to create a text mining process for qualitative
researchers using a simple design science method. We developed a five-stage process and tested it using a
small-scale experiment. The experiment demonstrated that the process was usable, but that most of the
issues that were identified through literature search became significant problems in use.
We asked the questions
 how can a low cost, low tech Internet text retrieval and analysis process be conducted to facilitate
qualitative research?
 what are the advantages and disadvantages of such an approach when compared to traditional
ways of collecting and analysing qualitative data?
In response to our first research question, we note that the literature we studied concentrated on the text
analysis techniques, but often assumed that the other stages were trivial not our experience. In particular
the problematisation and interpretation stages became difficult in the context of research, where open
questions are seldom encouraged and interpretations must be carefully justified. In understanding the
advantages and disadvantages of using such a process (RQ2) we acquired the learning that our labour
intensive style of (qualitative) research was limited to pitifully small quantities of well-targeted data, and was
extremely difficult to scale up. It was relatively easy to acquire a large volume of text from the Internet,
however there was a clear trade off between quantity and quality. Our traditional practice provided the data
we needed for analysis, but retrieving text from the Internet presented many additional problems. Whereas
we understand the norms for qualitative analysis, we had to make many relatively insecure choices with the
automated text mining tools, and lacked confidence that those choices would stand up to the scrutiny of
experience reviewers. There are no established standards for conducting this kind of research that simplify the
many methodological choices, and relatively few examples to follow. In addition there is little existing
methodological help for combining quantitative and qualitative text analysis, apart from the general precepts
of multi-method research.
Despite the difficulties, we expect the marriage of text mining with qualitative research to be a productive one
in the future, and made inevitable by current trends. Two conditions for progress are the wider availability of
better low cost software (this is expected to improve rapidly), and the emergences of some generally accepted
methodological standards and procedures for text mining as research. Some avenues for future
methodological research are:
 using text mining to suggest patterns in a large text corpus combined with selective or sampled
qualitative investigation of the patterns (the research style of our example)
 using information extraction techniques to understand ontological structures of text
 using natural language analysis techniques to understand semantic structures
 using automatic text translation to enable cross cultural qualitative studies
 automatically summarizing a large text corpus to provide a manageable amount of text for
qualitative analysis
 combining sentiment analysis with qualitative evaluation
 developing automatic question answering as a vehicle for hypothesis testing
The methodological challenges involved are considerable, but both text mining and qualitative research can be
richer as a result.
References
Abrahams, A. S., Fan, W., Alan Wang, G., Zhang, Z. J. & Jiao, J. (2014) An integrated text analytic framework for product
defect discovery. Production and Operations Management.
Alberts, D. S. & Hayes, R. E. (2002) Code of Best Practice for Experimentation. In: DoD Command and Control Research
Program, Office of the Assistant Secretary of Defense. Washington, D.C.
Alberts, D. S. & Hayes, R. E. (2005) Campaigns of Experimentation: Pathways to Innovation and Transformation. In:
Command and Control Research Program, Office of the Assistant Secretary of Defense Washinton. D.C.
Blei, D. M. (2012) Probabilistic topic models. Communications of the ACM, 55(4), 77-84.
Chau, M. & Xu, J. (2012) Business intelligence in blogs: Understanding consumer interactions and communities. MIS
Quarterly, 36(4), 1189-1216.
Chen, H., Chiang, R. H. & Storey, V. C. (2012) Business Intelligence and Analytics: From Big Data to Big Impact. MIS
Quarterly, 36(4), 1165-1188.
Choi, D., Ko, B., Kim, H. & Kim, P. (2014) Text analysis for detecting terrorism-related articles on the web. Journal of
Network and Computer Applications, 38, 16-21.
Cui, W., Liu, S., Tan, L., Shi, C., Song, Y., Gao, Z. J., Qu, H. & Tong, X. (2011) Textflow: Towards better understanding of
evolving topics in text. IEEE Transactions on Visualization and Computer Graphics, 17(12), 2412-2421.
Davenport, T. H. & Harris, J. G. (2007) Competing on analytics: The new science of winning. Harvard Business Press,
Brighton, MA.
Delen, D. & Crossland, M. D. (2008) Seeding the survey and analysis of research literature with text mining. Expert Systems
with Applications, 34(3), 1707-1720.
Dou, W., Yu, L., Wang, X., Ma, Z. & Ribarsky, W. (2013) Hierarchical topics: Visually exploring large text collections using
topic hierarchies. IEEE Transactions on Visualization and Computer Graphics, 19(12), 2002-2011.
Gandomi, A. & Haider, M. (2015) Beyond the hype: Big data concepts, methods, and analytics. International Journal of
Information Management, 35(2), 137-144.
He, W., Zha, S. & Li, L. (2013) Social media competitive analysis and text mining: A case study in the pizza industry.
International Journal of Information Management, 33(3), 464-472.
Hevner, A. R., March, S. T., Park, J. & Ram, S. (2004) Design science in Information Systems research. MIS Quarterly, 28(1),
75-105.
Ikeda, K., Hattori, G., Ono, C., Asoh, H. & Higashino, T. (2013) Twitter user profiling based on text and community mining
for market analysis. Knowledge-Based Systems, 51, 35-47.
Kaiser, C. & Bodendorf, F. (2012) Mining consumer dialog in online forums. Internet Research, 22(3), 275-297.
Kosala, R. & Blockeel, H. (2000) Web mining research: A survey. ACM Sigkdd Explorations Newsletter, 2(1), 1-15.
Krippendorff, K. (2004) Content Analysis Sage, Thousand Oaks.
Lu, B., Ott, M., Cardie, C. & Tsou, B. K. (2011) Multi-aspect sentiment analysis with topic models. In: Data Mining
Workshops (ICDMW), 2011 IEEE 11th International Conference on, 81-88. IEEE.
Miles, M. B. & Huberman, A. M. (1994) Qualitative data analysis (2nd edition). Sage, Thousand Oaks.
Moon, S., Park, Y. & Seog Kim, Y. (2014) The impact of text product reviews on sales. European Journal of Marketing,
48(11/12), 2176-2197.
Moro, S., Cortez, P. & Rita, P. (2015) Business intelligence in banking: A literature analysis from 2002 to 2013 using text
mining and latent Dirichlet allocation. Expert Systems with Applications, 42(3), 1314-1324.
O'Shea, M. & Levene, M. (2011) Mining and visualising information from RSS feeds: a case study. International Journal of
Web Information Systems, 7(2), 105-129.
Pang, B. & Lee, L. (2008) Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2), 1-
135.
Romano, N. C., Donovan, C., Chen, H. & Nunamaker, J. F. (2003) A methodology for analyzing web-based qualitative data.
Journal of Management Information Systems, 19(4), 213-246.
Rose, J., Jones, M. & Furneaux, B. (2016) An Integrated Model of Innovation Drivers for Smaller Software Firms.
Information & Management, 53(3), 307-323.
Rose, J., Markfoged, K. & Andersen, J. L. (2010) Can design science be used for design? Proceedings of the Fifth
Mediterranean Conference on Information Systems, Tel Aviv, AIS Association.
Vaishnavi, V. K. & Kuechler, W. (2015) Design science research methods and patterns: innovating information and
communication technology. CRC Press.
Vitolo, C., Elkhatib, Y., Reusser, D., Macleod, C. J. & Buytaert, W. (2015) Web technologies for environmental big data.
Environmental Modelling & Software, 63, 185-198.
Xiang, Z., Schwartz, Z., Gerdes, J. H. & Uysal, M. (2015) What can big data and text analytics tell us about hotel guest
experience and satisfaction? International Journal of Hospitality Management, 44, 120-130.

Appendix 1. sample blog download:

15 Techno Social Blog Techno Social Blog 15 http://rajasekar.net/page/3/
Predictive Analytics-A future Insight of Data Analysis February 9, 2008 Rajasekar Predictive analytics
encompasses a variety of techniques from statistics and data mining that analyze current and historical data to
make predictions about future events.Such predictions rarely take the form of absolute statements, and are
more likely to be expressed as values that correspond to the odds of a particular event or behavior taking
place in the future. In business, predictive models exploit patterns found in historical and transactional data to
identify risks and opportunities. Models capture relationships among many factors to allow assessment of risk
or potential associated with a particular set of conditions, guiding decision making for candidate transactions.
It is been used with the applications involving CRM,Cross-Selling,Direct marketing ,Collection analytics not only
that even helps to detect Fraud detection in credit card Apps. The statistical techniques used in Predictive
Analytics are as follows Regression Techniques Linear Regression Model Discrete choice models Logistic
regression Time series models Apart from these Statistical Techniques there are some Machine learning
techniques are used such as Neural Networks and k-nearest neighbours The tool used to help with the
execution of predictive analytics are SAS, S-Plus, SPSS and Stata and For machine learning/data mining type of
applications, KnowledgeSEEKER, KnowledgeSTUDIO, Enterprise Miner, GeneXproTools, Clementine, KXEN
Analytic Framework, InforSense are some of the popularly used options. WEKA is a freely available open-
source collection of machine learning methods for pattern classification, regression, clustering, and some
types of meta-learning, which can be used for predictive analytics. Recently Business Objects has announced a
partnership with SPSS , a worldwide provider of predictive analytics software, announced the companies have
entered into an original equipment manufacturer agreement in which Business Objects will offer its customers
the ability to use SPSS predictive analytics data mining technology as part of the market-leading Business
Objects TM XI platform. Users of Business Objects XI with predictive analytics data mining technology will be
able to leverage business predictions to make more informed decisions that can help generate revenue,
control expenses, and mitigate risk. Today SAS, the leader in business intelligence, has significantly enhanced
its award-wining SAS Enterprise Miner, SAS Text Miner, and SAS Forecast Server software, bringing predictive
analytics to their highest level yet. The newest release of SAS Enterprise Miner improves productivity through
added interactive advanced visualization and new analytics. Fifteen new analytical tools improve the resulting
predictive models, which can mean significant savings for customers with proactive marketing departments
such as in retail or banking.With innovative new modeling algorithms, including gradient boosting, partial least
squares and support vector machines, SAS Enterprise Miner users can build more stable and more accurate
models and thus make better decisions faster and with more confidence. Source:Wikipedia and SAS,Business
Objects Read More
Social Physics, Crowdsourcing and Multicultural Collaborative
Research Practice in the Social Sciences: E Pluribus Unum?
David A.L Coldwell

School of Economic and Business Sciences, University of the Witwatersrand, Johannesburg
david.coldwell@wits.ac.za
Abstract: The possibility of there being an investigator with the knowledge and ability to understand, let alone digest the
data embraced by the boundaries of social scientific disciplines seems very remote. However research investigators in the
human sciences might in their quest for greater validity and wider generalization in their concepts and theories, adopt a
different scientific perspective in addition to its currently most prevalent hypothetico-deductive approach. The proposed
approach made in the paper corresponds more to Baconian-type methods and those adopted by Darwin in his theory of
evolution and involves the use of large data sets of the kind made available by the methodological approaches of social
physics and crowdsourcing. The paper suggests that large data sets can be analyzed effectively in multi-disciplinary, cross-
cultural collaborative research contexts, and a case study illustrating a collaborative research process is described in detail
to demonstrate this point. The paper maintains that through the combined utilization of scientific approaches and large
data gathering techniques made available through new technologies, it may be possible to generate comprehensively valid
empirical and theoretical wholes across the human sciences.
Keywords: Social Physics, crowdsourcing, multicultural, multidisciplinary, collaborative research, social sciences.
1. Introduction
The possibility of there being an investigator who has the knowledge and ability to understand, let alone digest
the data embraced by the boundaries of social scientific disciplines seems very remote. Not only has the
amount of information that has become available today in general terms expanded exponentially, it has
occurred to the extent that it has become impossible for any single human being to digest and use it
intelligently. This has occurred in scientific and social scientific disciplines themselves such that scientists today
find themselves knowing more and more about less and less (Senge,1990) as their data bases expand and
greater specialization becomes imperative.
In effect, this information and data explosion alone has made any single research investigator capable of doing
effective research across the human sciences beyond the realms of human capacity. In spite of this there has
never before, perhaps in the history of scientific endeavor, been a time demanding generalist investigators
who understand cross-disciplinary knowledge and how to use it effectively, both in developing theories and in
practical implementations, than there is today. A general understanding of social scientific problems needs a
movement away from its current knowledge silos, yet paradoxically such silos are becoming increasingly
evident among research investigators who struggle to keep abreast of the developments in their own
disciplines.
The research question the paper attempts to answer is: how can the social sciences and business research
generate multidisciplinary and multicultural valid theoretical structures from generalizable data? To this end
the paper aims to contribute to the social scientific methodological literature by indicating that social science
research investigators should adopt a collaborative multidisciplinary and cross-cultural approach which
embraces the collection and analysis of large data sets to determine communalities and theoretical structures
arising from it.
The paper takes the following form. The second section gives the extra-disciplinary research problem’s
background by outlining the information explosion, the effect this has had on discipline specialization and the
development of new sub-disciplines and the ‘silo effect’ on knowledge generation that this has generated. The
problem of ‘methodological divides’ and the ‘pecking order of scientific rigor,’ both within and between
specific human science disciplines on the possibility of multidisciplinary collaborative research is also described
in this section.
ISSN 1477-7029 17 ©ACPIL
Reference this paper as: Coldwell D, “Social Physics, Crowd Sourcing and Multicultural Collaborative Research Practice in
the Social Sciences: E Pluribus Unum?” The Electronic Journal of Business Research Methods Volume 15 Issue 1 2017,
(pp17-28) available online at www.ejbrm.com
The third section discusses the practical intra-disciplinary research problem faced by social scientists who
aspire to generate multi-disciplinarily-valid knowledge across the human sciences and aim at creating cross-
disciplinary theoretical structures that offer practical solutions to multi-facetted social problems, brought
about by intra-disciplinary problem of methodological divide. A problem that compounds the silo-generating
effect arising from the information explosion discussed in the second section.
The fourth section presents a core contribution of the current paper by highlighting the author’s experiences
of initiating a multinational and multidisciplinary group consisting of a group of researchers combined in
collaborative research across the human sciences. It is proposed that a cross-cultural, multidisciplinary,
collaborative research approach can aid in counteracting the effects of the extra-disciplinary problems of the
information explosion and multidisciplinary silos and the intra-disciplinary problem of methodological divides.
The fifth section of the paper considers a possible way forward in the human sciences towards the generation
of valid knowledge by proposing a combination of the data gathering approaches of social physics and
crowdsourcing one the one hand, with multi-national and multidisciplinary collaborative research groups on
the other. It is suggested that by advocating the observational-deductive paradigm used in evolutionary
biology rather than depending on the inductive hypothesis-testing ‘falsificationist’ approach which advocates
testing of hypotheses amenable to falsification (Popper, 1992), that currently prevails in positivistic social
science and, most importantly, by acknowledging the significance of qualitative data, the process of theory
building and practical problem-solving could be better accomplished. The sixth and final section of the paper
concludes with a brief summary of the main arguments presented.
2. Background to the problem: information deluge, specialization and methodological

divides
There can be little doubt regarding the exponential increase in information as a whole and within academic
disciplines in particular and through this, the development of new disciplinary areas. Although there is
surprisingly little research dealing directly with this aspect, it is reasonably safe to say that the information
explosion within particular disciplines, along with the discovery of different empirical problem areas that
demand scientific endeavor that spring largely from this, has demarked an increasing number of new
disciplines and sub-disciplines over the years. Of course it is also true to say that the founding of a particular
discipline or sub-discipline, has led to new areas for information mining and, through this, its own information
sourcing demands and requirements. According to a recent Berkley research project document (Lyman and
Varian 2003) the storage of information (paper, film, magnetic and optical) has been expanding at the rate of
over 30% per annum since 1999 and that there has been a dramatic increase from 2001-2003 in the storage of
new information in every storage medium except film. The expansion in paper information from 2000-2002
was estimated as 1,634 Terabytes (a Terabyte being the equivalent of 1012bytes) which requires around
788million trees to produce the world’s annual paper demand.
http://www2.sims.berkeley.edu/research/projects/how-much-info-2003/. Partly as a result of this expansion
in information and the need to mine new information because of the expansion of sub-disciplines within
disciplines, the discipline of Economics alone now consists of around 25 sub-disciplines. Some sub-disciplines
are of recent origin (e.g. Behavioral economics, Complexity economics and Bioeconomics) and in some cases
fast becoming, partly as a result of burgeoning information available to them, new disciplines in their own
right.
Rosvall and Bergstrom (2010) mapped more than 35 million citations between articles that appeared in
around 7000 scientific journals. Their analysis of information flows between researchers worldwide showed
that neuroscience has developed from being an interdisciplinary research topic to being a scientific discipline
of its own and become the fifth largest field in the sciences ( after molecular and cell biology, physics,
chemistry and medicine). Rosvall and Bergstrom (2010, p.2) write: “Only in the last decade has this change
come to dominate the citation structure of the field and overwhelm the intellectual ties along departmental
lines”.
Although the above data relates to science and engineering rather than the sprawling disciplines and sub-
disciplines that make up the social sciences, making them much harder to plot their development, the
foregoing brief analysis clearly suggests that not only has information expanded considerably recently, but this
David A.L Coldwell
expansion has made possible and in many cases been an important factor, in the development of new sub-
disciplines within existing ones which have, in some cases, become separate disciplines in their own right.
The general trajectory of information proliferation and disciplinary expansion in the social sciences has made
the possibility of social scientific research investigation extraordinarily difficult. The problem is exacerbated by
the traditions of academic disciplinary exclusivity and chauvinism. This aspect is discussed in more detail in the
following section.
2.1 Disciplinary discordance between and within
No scientific discipline operates free of internal divisions and external detractors. Even physics has been
criticized by mathematicians for the imprecision of some of its elementary assumptions. (Spivak, 2010). The
discordance between the natural sciences and social science is even more pronounced than within natural
scientific disciplines per se. In this regard, Duffy, Guertal and Muntifering (1997, not paginated) suggest that:
“For most natural scientists research follows a set course: formation of a hypothesis, design of an experiment,
collection of experimental data, and analysis of data using a limited number of statistical techniques. Journal
articles in the natural sciences tend to be short and factual, with little effort expended on justification of the
methods used. By contrast in economics research experiments per se are seldom conducted. Most data are
second-hand, and methods of analysis are numerous and often highly contested in the field”.
Within and between the social sciences themselves discordance and disagreement run much deeper and is
more widespread. Within sociology, for example, a deep seated, long established and perhaps unbridgeable
(at least among the more vociferous protagonists on both sides) divide exists between positivism which
maintains “…that positive knowledge is based on natural phenomena and their properties and relations as
verified by the empirical sciences” (Merriam-Webster, 2016) and phenomenology, which focuses on free-of-
researcher-pre-conceptualizations descriptions of phenomena as consciously experienced by an individual
(Biernel and Spielberg 2015).
Within the wide disciplinary ambit that embraces the social sciences the schism is perhaps most keenly felt
between economists and the other social sciences. Comte may have regarded sociology as the ‘queen of the
social sciences’ (Turner, Beeghley and Powers, 2012) but today most academics would acknowledge (if not
accept) that economics (perhaps by virtue of the fact that it is the only social science that attracts a Nobel
prize and that therefore some of the best social scientific brains tend to gravitate towards it) is dominant in
the ‘pecking order’ of the social sciences. Certainly I think that most economists regard themselves as pre-
eminent in the social science disciplinary ranks and tend today to see other business science disciplines (e.g.
Management) as riddled with problems of endogeneity in their explanations and limited in the validity of their
findings by virtue of the small data sets generally used.
Although offered as a stepping stone towards multi-disciplinarily corroborated social science research in the
case study to be described in the following section, it must be recognized that there are multifaceted and wide
ranging difficulties involved in conducting multi-disciplinary and multicultural collaborative social science
research. Perhaps the most prominent and tenacious ‘costs’ in such undertakings spring from three main
sources: methodological disagreements, cultural factors and limited generalizability.
2.1.1 Lack of methodological consensus among practicing social scientists
Lack of methodological consensus in the social sciences is legend and ongoing and can only be dealt with very
briefly here. Its origins are multiple. Epistemological and ontological differences of opinions among social
scientists have resulted in a plethora of different methodological approaches. Possibly the most far-reaching
and intractable divide resides between the approaches of positivism on the one hand and phenomenology, on
the other. Social scientists are sometimes persuaded by specific epistemological and ontological arguments,
sometimes brought up under the traditions of one or other perspective, or sometime locked into a particular
approach (usually qualitative) by a lack of mathematical/statistical knowledge and/or aptitude.
2.1.2 Cross-cultural social scientific communities-of-practice and the problem of reflexivity
An example of a cross-cultural collaborative research process is described by Easterby-Smith and Malina
(1999). In this case the investigation deals with the quality of business relationships between the United
Kingdom and China and constitutes a singular project. Easterby-Smith and Malina (1999) describe the reflexive
process that occurred in a ‘one-off’ cross-cultural research project conducted by academics in the U.K and

China with homogenous disciplinary backgrounds. Jain, Hallensleben and Manger (2013) suggest that
reflexivity can be considered a dialectical concept. A dialectical understanding of reflexivity includes that the
implementation of reflexive structures in cross-cultural communities-of-practice which “may cause dialectic
counter-movements in the shape of defensive reactions” (Jain, Hallensleben and Manger, 2013, p.9). These so-
called deflexive defensive reactions are distinguished from reflexivity in the following manner by Jain,
Hallensleben and Manger (2013, p.10): “while reflexivity produces difference, deflexivity basically means the
production of identity, unambiguity and firmness by means of structural momentum and structural violence.
But neither is… reflexivity merely “positive” nor is deflection/deflexivity fully “negative”. Both carry
“productive” and “destructive” elements. The perpetuated self-questioning and the openness invoked by
reflexivity are counter-productive when it comes to the practical appliance of learning experience and new
ideas, i.e. innovation. This is exactly the strength of deflexivity”. When it comes to cross-cultural collaborative
research the concepts of reflexivity/deflexivity offer a useful heuristic to describe the balance needed in such
research communities of practice if, on the one hand, their cultural diversity is to be effective in opening up
practitioners’ creativity and the development of new bodies of knowledge reflexivity must be allowed full rein.
But, on the other hand, deflexive responses are required in the sense-making, congruence-seeking and
consensus knowledge building of the community.
2.1.3 Lack of generalizability of social scientific concepts and theories
There is little doubt that a major difficulty experienced by the social sciences, perhaps with the exception of
Economics, is that data sets on which concepts and theoretical edifices are built are generally quite small. A
good recent example of this in Economics is the research by Bloom and Van Reenen, (2007; see also Bloom et
al, 2012) Economists working at Stanford University and the London School of Economics respectively, who
have studied the effects of management in the performance of more than 10,000 companies in 20 countries
by applying econometric methods. Although this research is wide scale and incorporates large data sets from
multicultural contexts and is collaborative to an extent, it lacks a multidisciplinary and thus multi
methodological validity and generalizability.
In the analysis of qualitative data, small data sets are of course the norm, but even here replication is curtailed
by the fact that such data sets tend to be very small and difficult to re-engage because of this. There have been
attempted solutions round this problem such as meta-analytical research and the like, but such studies are
limited by their historical periodicity and the diversity of contexts they generally cover. Also, it is unlikely that a
collection of relatively small samples gathered at different points in time in different contexts and used to
generate a larger combination in meta-analyses, is able to produce valid up-to-date generalizations in the
manner large data sets taken and analyzed at one point in time can.
3. Bridging the methodological divide: the place of multidisciplinary multinational

collaborative research groups
A breathtaking example from the natural sciences of the success of small size multidisciplinary group
collaborative research is presented by the Californian Institute of Technology, an institution that is now 123
years old with 57 recipients of the US National Medal of Science and 32 Nobel laureates among its faculty and
alumni (Baty, 2014). There are 5 Nobel Prize winners on its current staff. Caltech’s small size is considered to
be the central, core reason for its fabulous global success in scientific research. Caltech’s small size has made it
obligatory for it to operate through interdisciplinary collaboration. An example of the interdisciplinary ethos of
the institution is given by the Division of Biology (founded in 1928) which was transformed in 2013 into a new
Division of Biology and Biological Engineering. This change occurred after the so-called “organic drift” of the
Division of Engineering and Applied Science’s bioengineering department into synthetic biology, which focused
on manipulating biological materials. Of course another clear reason for its success has been its highly
selective approach to recruiting the best staff available, freeing them from administrative burdens and
providing them with excellent salaries, research funding and equipment (Baty, 2014).
Interdisciplinary collaborative research in the social sciences is different from the ideal type presented by
above example of Caltech in at least three main ways:
• Lack of methodological concensus among practicising scientists

• Cross-cultural social scientific communities-of-practice differences in values and social priorities:
• Limited generalizability of concepts and theories
David A.L Coldwell
The benefits of multidisciplinary collaborative research groups are more evident in disciplines comprised of the
natural sciences perhaps because of the general consensus among scientists in these disciplines of what
comprises scientific method (Duffy, Guertal and Muntifering, 1997). This is not to say that differences in
methodology don’t exist, or that such differences do not create fundamental disagreements among natural
scientists. Clearly they do and perhaps are most keenly felt in the methods of evolutionary biology, where its
inability to create mathematically precise explanations of natural selection and the theory’s inadequacies as
regards prediction of future life forms, is regarded with disfavor among Physicists and Chemists. How these
problems peculiar to the social sciences might be successfully dealt with is discussed in more detail in the
following section.
4. Benefits of multi-disciplinary and multicultural collaborative research: A case study of

a cross cultural multidisciplinary research group
Collaborative cross-cultural multidisciplinary research in the social sciences is not only possible, but can
generate enormous benefits in terms of the interdisciplinary richness of perspective and cross cultural validity
of the concepts and theories utilized. This aspect is described in some detail below in relation to the author’s
recent experiences of directing an international multidisciplinary collaborative research group: the Strategic
Foresight Research Group (SFRG) created for this purpose.
The SFRG was initiated in 2011 as a tool for research generation. Its purpose and function is to:
• Generate high quality research outputs through collaborative national and international research.
• Augment the research output.
• Offer fledgling researchers help to promote their research expertise.
• Offer junior researchers supervision for advanced degrees.
• Offer experienced international researchers the opportunity to work in collaborative research with
researchers in South Africa.
A major feature of the SFRG is that it offers junior researchers both the opportunity to develop their formal
research qualifications and to produce publishable research outputs through expert training and guidance.
The Strategic Forecasting Research Group research focuses on broadly defined strategic issues in business and
other formal organizations. Multidisciplinary and multinational research projects are aimed to investigate
management and organizational issues and to provide multicultural theoretical and empirical perspectives.
Research by the group covers aspects of international concern such as: governance, social responsibility,
responsible leadership, intercultural conflict management, research productivity, organizational citizenship,
entrepreneurship, teaching and learning.
The group involves around sixteen active researchers with roughly half based overseas. The group adopts a
matrix structure in the research process. Researchers take leadership roles for specific projects as they arise
based on personal interest and expertise. Research is funded by local and overseas grants and through
financial packages/allowances provided to individual group members by virtue of their graded research status.
Research meetings are held by national (SA-based) members of the group when project-related research
issues arise. Research workshops focus on communicating new multidisciplinary research ideas among
national members to build collaborative research teams for particular projects, and to decide on applications
for funding. Virtual workshops to decide on research outputs and collaborations for specifically targeted
conferences and journals are conducted electronically as required.
Easterby-Smith and Malina (1999) identify five steps in the collaborative research process. Harnessing
networks, the initial phase of group formation, consisting of informal meetings aimed at creating trust and
communication among team members. The next phase, focusing the project defines objectives of the research
and is a crucial aspect for applications for research funding. The U.K. research team, reported by Easterby-
Smith and Malina, (1999) focused on the research objectives and the economic utility of the project required
by the British funding agency. Chinese team members of the UK driven research project needed to modify the
research proposal to show its contribution towards the Chinese economic reform process. Easterby-Smith and
Malina (1999) report that at the beginning of the research collaborative process, team UK and Chinese
members realized that both sides saw the other’s research interest differently and attributed a “slightly less

positive, practical and commercial motives to the other national group” (Easterby-Smith and Malina 1999, p.
79). Over time it became obvious in collaboration between Chinese and UK researchers that funding agencies
of each country injected a political element to the project by stipulating specific practical outcomes. Accessing
data, the third step in the research process brought into sharp focus the status and expertise of the
researchers and their role in the research process. This differed quite markedly between the two nations and,
in some instances, contradicted internal perceptions of expertise and seniority of members themselves. For
example, a young female researcher was ignored by British interviewees whereas in China she was regarded
by Chinese interviewees as important. Also, field-work procedures differed between the two national research
groups regarding interview settings and empirical data collection procedures.
During the interpretation or making sense out of the data in the fourth phase, reaching a satisfactory
consensus took substantial time. This was partly due to differences between the Chinese and U.K. research
teams’ ontologically-driven methodological approaches. U.K. researchers were interested in qualitative
empirical material, while their Chinese counterparts focused more on finding similarities and associations
using quantitative data. In line with Nonaka and Takeuchi (1995), Easterby-Smith and Malina (1999) explain
this by ‘dualistic’ (search for differences), vis a vis ‘holistic’ (search for similarities) distinction in researcher
perspectives. The interpretation phase clearly showed differences in UK and Chinese research perspectives,
educational traditions and methodological perspectives. There was a need to discuss and negotiate similarities
and differences in interpretations in the research teams to reach mutually acceptable consensus.
In the fifth, writing and dissemination phase of the research process team members decided “that all data
collected on the project should be considered equally the property of every member” (ibid., p. 81). Deep
insights of the qualitative empirical data obtained by a Chinese and U.K. researcher enabled them to publish in
prominent refereed journals. Also, a number of articles were published with differing co-authorships reflecting
the substantive collaborative effort in the research process.
The Easterby-Smith and Malina (1999) phases of the UK-Chinese collaborative research process is used to
classify the SFRG research project processes in Table1.
Table1: National and multinational research collaboration: specific phases, descriptors and research outputs
(Adapted from Coldwell and Fried, 2014).
National and Phase 1: Phase 2: Phase 3: Phase 4: Phase 5: Phase 6:
multinational Harnessing Focusing Accessing Interpretation Writing and Research
research networks research data dissemination outputs
collaborative project
teams
South Africa Interdisciplinary Projects arose Group Project Writing of National and
(Founding foundations of from mutual research leaders of research international
institution with network built research division of specific product conferences
national on: interests and labor research conducted by and national
multidisciplinary i) Personal commonalities responsibilities project project leader. and
research team contact. in research in: interpret the Project group international
consisting of: ii)Academic focus i) Data data from data members journal
accounting, and research collection analysis involved in articles with
auditing, interest instrument, ii) secondary different
governance, iii) Research Sample research and management
finance, group design and iii) project science foci.
economics, formation Data ‘housework’
marketing, administration e.g. editing.
management and collection referencing
and HRM) and capture. and formatting
South Africa- Network from Specific Project group Three project Collaboration Project
Canada i), ii) and iii) project arose Division of collaborators, between SA currently in
arising from a from labor based 2 in SA and 1 and Canada sample
specific conference on expertise in Canada project acquisition
conference paper i) ii) and iii) members and
meeting in presentation accessing
Johannesburg data stage
David A.L Coldwell
UK, South Network from Specific Group Division Project Collaboration Conference
Africa-Germany i), ii) and iii) project arose of labor based collaborators between SA papers and
arising from from visiting on expertise i), consisting of and German international
Summer professorship ii) two members, researchers journal
School (Open and iii) 1 in Germany article
originally held University, and 1 in SA
in Chemnitz Milton
Keynes) and
Summer
School in
Chemnitz
South Africa- Network from Specific Group Division Project Collaboration Chapter in
Sweden i), ii) and iii) project arose of labor based collaborators between SA book
from invitation on expertise i), 1 in Sweden and Swedish
to write a ii) interest and 1 in SA researchers
chapter in and iii).
Knowledge
Management
research
handbook
book
South Africa- Network from Specific Group Division Project Collaboration Conference
United States of i), ii) and iii) project arose of labor based collaborators between SA paper and
America arising from from on expertise i), 1 in USA and and USA International
meeting at a conference ii) interest 3 in SA researchers journal
conference in paper and iii). articles
Johannesburg presentation
South Africa- Network from Specific Group Division Project Collaboration Chapter in
New Zealand i), ii) and iii) project arose of labor based collaborators between SA book
arising from an from invitation on expertise i), consisted of 1 researcher
invited to contribute a ii) in and New
personal visit to chapter in a and iii). SA and editor Zealand Editor
SA during book on of book in
sabbatical corporate New Zealand
leave social
responsibility
South Africa – Network arose Projects arose Group Division Multiple National and Conference
United Kingdom from from of labor based projects’ international papers
i), ii) and iii) supervision of on collaboration collaboration national and
PhD supervisor doctoral expertise i), ii) 4 SA and 1 in between SA international
of a research degree of one and iii). UK and UK journal
group member. member of articles
SA conference. the research
Personal visit Group and
to SA and conference
meeting of UK papers
professor.
Invitation to
join UK
research
group. UK
conference
Network arose Project arose Group Division Project Collaboration Project

South Africa- from ii) from article of labor based collaborators between SA temporally
Israel Academic and publication in on 3# in South and Israel suspended
research an expertise i), ii) Africa and 1 in researchers
interest international and iii). Israel
journal on a (currently not
research topic a member of
of mutual the SFRG)
interest
Table I clearly indicates that the SFRG has been successful in conducting collaborative research that is
culturally sensitive and multidisciplinary. Recently the South Africa-Israel has had to be suspended for political

reasons. ‘Multidisciplinary’ in this context means disciplines that are all within the general ambit of business
studies and commerce. Differences between research methods employed by, for example, accountants and
financial management researchers are quite distinct from those adopted by Management and Human
Resources Management, particularly regarding the importance of qualitative data. ‘Multicultural’ in this
context generally refers to research projects located in the specific countries involved. However, the principle
of multicultural research collaboration and its practical possibility are clear from the recent research work of
the SFRG involving different countries. For example, collaborative work involving researchers based in
Germany, Sweden, South Africa and the United States of America, led to a research outputs (Coldwell and
Fried, 2012) and (King, Joosub and Coldwell, 2013) that were multidisciplinary and cross-culturally aware.
The problem of reflexivity in the interpretation phase of collaborative research work discussed earlier in the
paper is contained to a significant extent in the Strategic Foresight Research Group (SFRG) in at least three
specific ways.
1. The research collaboration performed by the SFRG is multinational and involves at its present level
of development collaboration between South African, German, Swedish, Canadian American, British.
New Zealand, Australian and Israeli - based academics.
2. The research projects conducted by the SFRG are on-going, with completed projects being replaced
by completely new ones or derivatives of older ones. Different projects do not necessarily involve
the same academic personnel from the same countries and,
3. The research team combines researchers across a wide spectrum of the Management discipline and
includes Accountants, Auditors, Marketers and specialists in Management, HRM, Finance and
Governance.
Disciplinary diversity and multiculturalism of the research meant that various perspectives emerged, and
although particular individuals led particular projects in line with their disciplinary emphases, the common goal
of publishable products ensured that overall consensus was achieved.
The multi-disciplinary and multicultural collaborative research process in the SFRG clearly shows that the
reflexive process among researchers is overridingly a positive one, where differences in disciplinary
perspectives, methodological approaches and cultural and educational backgrounds have blended to generate
bodies of new knowledge. Despite this, extant social scientific research is hampered the lack of generalizability
of its findings and emanates from the small-scale of the data bases often used. The following section discusses
this aspect in more detail.
5. A methodological way forward? The place of large data sets in social science
investigation: social physics and crowdsourcing
Evolutionary theory was partly developed from a large series of acute flora and fauna observations made by
Darwin who writes: “During some part of the day I wrote my journal and took much pains in describing
carefully and vividly all that I had seen and this was good practice”. (Darwin, 2009, p. x).
Such observations led to the development of Darwin’s theory of natural selection, although it is also suggested
that Darwin had already the notion that species gradually became modified and that it had ‘haunted’ him
(Penny, 2009) before his observations were able to corroborate this in any systematic way. But before these
‘haunting hypothesis’ were merged with his observations and classification of large scale biological data, it was
observation pure and simple similar to that Popper (1963. p.46) ridicules: “The belief that we can start with
pure observation alone, without anything in the nature of theory is absurd; as may be illustrated by the man
who dedicated his life to natural science, wrote down everything he could observe, and bequeathed his
priceless collection of observations to the Royal Society to be used as inductive evidence”. The approach to
scientific discovery recommended by Popper (1992) is quite different and emphasizes an ex- ante, hypothesis-
testing method used in positivistic social scientific research. However, it is proposed that the large data set,
observational approach advocated by Darwin could provide the basis on which the social sciences can produce
valid generalizable knowledge and dissolve the methodological divide.
Although mixed methods are becoming more prominent research tool, they are still hampered by the fact that
they tend to deal with relatively small data sets and are limited to a mono-cultural and mono-disciplinary
David A.L Coldwell
perspectives. However the current paper suggests that through the initial accumulation of large data sets of
the kind made available by methodological approaches of social physics and crowdsourcing, it is possible to
use both quantitative and qualitative approaches in subsequent analyses to generate comprehensively valid
empirical and theoretical wholes. This approach would also require ‘discipline bridging’ of the kind suggested
by de Kok (2013). In this regard de Kok (2013. p.12) states: “Real understanding between academics from
different disciplinary backgrounds requires the translation of each other’s concepts (Bevan, 2000) and ‘active
attempts at clear communication’ (de Kok, Hussein and Jeffery, 2010). The onus is on social scientists to use
language and concepts that can render their ideas accessible and thus engage scholars from other disciplines”.
And, one might add, scholars with diverse cultural perspectives.
Such large data sets could be regarded as providing the quantitative, non-verbal ‘bread crumbs’ (Pentland,
2014) of human behavior through recent developments in social physics, while the qualitative, verbal ‘loaves’
arising from large data sets would emerge from ‘crowdsourcing’. Thus large data sets, Social Physics and
crowdsourcing are seen as methodological alternatives that present an opportunity for valid research across
the social sciences, while at the same time providing large scale data for collaborative research teams to
generate comprehensively valid accounts of social scientific phenomena. Also, to attend to intercultural
differences in the application and meaning of specific social scientific concepts, such collaborative research
teams would be comprised of multi-national scholars.
The origin of the term ‘social physics’ is attributed to Auguste Comte in his book “Philosophie positive” (2009).
However, a recent adaptation has come through the work of a physicist Ball (2004) in his book entitled:
“Critical mass”. Ball (2004) regards a people as human particles which are then studied in computer
simulations. Critical mass is regarded as the statistically averaged behavior of human particles which taken
together with large data set observations, offer a critical mass typology of aspects of human behavior. Not
unsurprisingly, Ball has met with some fierce criticism from, in particular Fuller (2005), an American sociologist
who regards the idea of social physics as an affront based on a defunct reductionist (i.e. inappropriately
reducing complex phenomena to simple ones) conception of social science that died in the late 19 th century.
Despite this and other attacks on the ‘reductionist’ assumption of regarding humans as atoms or molecules, a
recent more sophisticated revitalization of the idea that the study human behavior ‘en masse’ can help
develop theories of behavior from large data sets has been presented by Pentland (2014). Pentland (2014)
suggests that human ideas of ‘common sense behavior’ are developed in our social interactions and the social
ideas and notions that arise from our social networking behavior. New technology such as that afforded in the
widespread use of smart phones and the internet, have led to a digital stream of ‘bread crumbs’ that can be
studied and analyzed. Pentland (2014) suggests that it is possible to discover patterns of information exchange
in social networks without knowing its detailed content, that inform on specific social behavioral patterns that
allow accurate predictions to be made of probable specific human behavior. Pentland (2012) suggests that
social physics helps understand how ideas flow from one individual to another through social learning to
create norms in behavior and shows how new ideas are adopted and transformed into changes in social
behavior. Pentland (2014) also suggests that the traditional social sciences (perhaps with the exception of
Economics) have always tended to be data poor and have not been able to make valid generalizations and
predictions. Social physics by accessing large data sets is better able to deal with a wide spectrum of social
issues. Whatever ones views on reductionist explanations of Social Physics, there is little doubt that the use of
large data sets can improve the accuracy of social scientific research and through the widespread use of
modern technology (currently mainly in the Western world but with increasing preponderance in the
developing world) among large groups of people, it becomes possible to trace and analyze digital streams
(social ‘bread crumbs’) to offer widely corroborated basic social data.
If social physics offers the quantitative, non-verbal ‘bread crumbs’ of social behavior on a wide scale, it might
be said that crowdsourcing can offer the qualitative verbal ‘bread’ from the study of large scale data sets.
Howe (2006, p.1) succinctly indicates how crowdsourcing originated: “Technological advances in everything
from product design software to digital video cameras are breaking down the cost barriers that once separated
amateurs from professionals. Hobbyists, part-timers, and dabblers suddenly have a market for their efforts, as
smart companies in industries as disparate as pharmaceuticals and television discover ways to tap the latent
talent of the crowd.”
Crowdsourcing can be regarded as a multi-functional technique for: soliciting mass informants’ views of
solutions to specific problems, suggest innovative ways forward and describe widely held attitudes and

perceptions and knowledge discovery through information (Brabham, 2008; 2013). The immediacy of crowd
sourced information allows the researcher to gather rapid data turnaround on a large scale with high
generalizability and validity potentiality. Of course this is not to suggest there are not potential disadvantages
and difficulties in crowdsourcing information of which information quality, research confidentiality and
response reliability, are considered the most serious (Stoyanova, 2010).
Page (2000) suggests that the diversity of the information garnered is a core advantage of crowd-sourced
material. Page (2000) has suggested from experiments conducted that diversity can and often does trump
ability. Under certain conditions, a random selection of crowd-sourced problem-solvers outperforms a
collection of the premier experts in the specific field largely because of the homogeneity of their thinking.
Experts were found to be better at finding solutions than the crowd but only with regard to highly focused
problems. Von Hayek (1945) perhaps was the first to note that every person has an advantage of others
because of the uniqueness of their personal experiences and information and that this information has
potential benefits to society.
In the field of business and specifically product innovation and marketing, crowdsourcing has been found to be
especially useful. Chesbrough (2003) cites the case of Proctor and Gamble who changed their approach to
innovation by crowdsourcing information on the premise that although they had 8,600 R&D scientists working
for them looking for such innovations there were 1.5 million outside. Far from being a method confined to the
social sciences, crowdsourcing has been used by neuroscientists. Thomas (2013) indicates that Seung of the
Max-Planck Institute devised a computer game called ‘EyeWire’ to help map the shape of actual eye neurons.
The neuron skeletons are pre-drawn and players of the game are required to color in the neurons. The hope is
to produce a map of a synapse that has never been mapped before and to find out how neurons detect
motion. Seung is currently writing a journal paper detailing new discoveries made from crowd-sourced
material obtained from the ‘EyeWire’ game. Both within the natural and social sciences the advantages of
crowdsourcing are becoming increasingly apparent and a respected scientific method. Buecheler et al (2010)
allude to the scientific methodological potential of crowdsourcing more in a recent article on open innovation
and collective intelligence. Buecheler et al (2010) refer to Malone, Laubacher and Dellarocas (2009) model
which uses the genome analogy to show the different aspects of a collective intelligence (crowd sourced ) task
which are attributed 4 basic “genes”: Who, Why, What, How.. The ‘Who’ aspect refers to the crowd or
hierarchy to be sourced, the ‘Why’ refers to the incentives for doing the collective intelligence task, which are
listed as money, love or glory. The ‘How’ refers to the structured process for obtaining the collective
intelligence of the crowd and the ‘What’ refers to the constituent of the crowd sourced material which might
be, inter alia: voting, averaging opinions/attitudes, innovative ideas, consumer behavior etc. Although not
specifically included in the model suggested by Malone, Laubacher and Dellarocas (2009), the ‘Where’ aspect
of context and culture is of particular importance in the social sciences for obtaining generalizable data in the
construction of valid concepts and theories. What is clear, however, is that crowdsourcing is a scientific
method that can lead to large scale data collection based either on specific ‘ex- ante’ hypotheses that the
researcher may wish to test; or, as in the case of innovation, a means of collecting ideas from detailed crowd
sourced information which, ‘ex post facto’, can be used to build testable theories and concepts.
In a very general sense, what social physics and crowdsourcing advocate is a movement towards large
inductive data ‘observations’ from careful scientific work and the subsequent development of theory ‘ex post
hoc’ and/ or ‘ex ante’ hypothetico-deductive process of testing particular hypotheses and away from smaller
data sets used in experimental and non- experimental situations. However this is not meant to suggest that
experimental and non-experimental smaller data set falsificationism has no place in social science. Moreover
the adoption of the Darwinian approach that includes detailed observations and documentation of large scale
data in the collection of social data made possible through modern technology and the methodological
approaches of social physics and crowdsourcing , does not mean that hypothesis testing of sections of general
findings allowed through large data sets is excluded. Large set data acquisition does not exclude piecemeal
research of segments of the large scale data collected utilizing quantitative hypothesis-testing, nor does it
exclude in-depth qualitative analyses ‘thrown-up’ by such large scale observation. In fact this was also
apparent in the original theory of natural selection (Weiner, 1994). Although there is clear evidence that
Darwin worked with a conjecture of a possible explanation for evolution in natural selection (Penny, 2009),
Darwin also worked on Baconian-type (Hesse, 1964) premise collecting much of his flora and fauna data
principles without a clearly formulated theory. It was this meticulously detailed observation that led Darwin to
present hypotheses of what had been observed and later his general theory of evolution. In his use of
David A.L Coldwell
meticulous observation methods, Darwin’s work could be said to be quite far removed from the Popper’s
notion of scientific knowledge creation based on ‘ex ante’ conjectures and refutations ‘ex post hoc’. In
crowdsourcing and social physics this element of collecting large scale data without necessarily having a
picture of what might emerge from it as in, for example, the collection of the collective intelligences of
innovative ideas sourced by neuroscientists and business organizations, can pay large dividends in social
scientific research work.
6. Summary and Conclusion

This paper has suggested that multidisciplinary, cross-cultural collaborative research groups ( de Kok, 2013)
and the use of large data sets in the manner advocated by the approaches of social physics and crowdsourcing,
can offer a way forward for social scientific research. Against the backcloth information expansion which has
led to increasing disciplinary specialization and the emergence of entirely new disciplines, the idea of a
research investigator(s) creating valid data across the social sciences is difficult to imagine. However,
multicultural and multidisciplinary research is both possible in the social sciences and has been shown to be
highly beneficial, despite difficulties that have emerged in its use. A major problem in social science research is
its general tendency to use smaller data sets which tends to limit the generalizability and validity of its findings.
Larger data sets such as those afforded through the approaches of social physics and crowdsourcing are seen
as providing a possible way forward when combined with a multidisciplinary and multicultural perspective in
collaborative research teams. Such large data sets could be initially analyzed through ‘pure observation’ with a
view to extracting communalities and associations that arise from this ‘ex-post facto’ in the manner Darwin
devised the of the theory of natural selection. It is envisaged that social scientific adaptations of this approach
can only be effectively conducted through multicultural and multidisciplinary collaborative research teams.
Within these larger data sets, it would also be possible to ‘hive off’ segments for piecemeal multicultural and
multidisciplinary qualitative, quantitative or mixed methods research with a view studying smaller units of the
whole in more detail. Practical implications and potential benefits of this general approach to the social
sciences and business research are manifold and include: more valid theoretical structures, more generalizable
knowledge and a breaking down of burgeoning disciplinary silos that seriously undermine scientific progress
and innovation.
References
Ball, P., 2004. Critical mass: how one thing leads to another. London: Heinemann.
Baty, P., 2014. Caltech: secrets of the world’s number one university. Times Higher Education 6 February [online] Available
at :< http://www.timeshighereducation.co.uk> [Accessed 6 March 2014].
Buecheler, T. Sieg, J.H., Fuchslin, R.M., and Pfeifer, R., 2010. Crowdsourcing, open innovation and collective intelligence in
the scientific method: a research agenda and operational framework. [online] Proceedings of the Alife XII
Conference, Odense, Denmark. Available at: <http://mitpress.mit.edu/> [Accessed 15 January 2014].
Bevan, P., 2006. Researching wellbeing across the disciplines: Some key intellectual problems and ways forward. ESRC
Research Group on Wellbeing in Developing Countries, WeD Working Paper, 25.
Biernal, W. and Spiegelberg, H., 2016. Phenomenology. [online] Encyclopedia Britannica,
<https://www.britannica.com/topic/phenomenology> [Accessed 20 October 2016]
Blair, A.M., 2010. Too much to know: Managing scholarly information before the Modern Age.
London: Yale University Press.
Bloom, N. and Van Reenen, J., 2007. Measuring and explaining management practices across firms and countries, The
Quarterly Journal of Economics, CXXII (4), pp.1351-1408.
Bloom, N., Genakos, C., Sadun, R. and Van Reenen, J., 2012. Management practices across firms and countries. [online]
National Bureau of Economic Research. <http://www.nber.org/papers/w17850> [Accessed 10 February, 2014].
Brabham, D. C., 2013. Crowdsourcing. London: The MIT Press.
Brabham, D. C., 2008. Crowdsourcing as a model for problem solving: An introduction and cases convergence. The
International Journal of Research into New Media Technologies, 14(1), pp.75–90.
Chesbrough, H.W., 2003. Open innovation. The new imperative for creating and profiting from technology. Boston:
Harvard Business School Press.
Coldwell, D.A.L. and Fried, A., 2012. Learning organizations without boarders? A cross-cultural study of university HR
practitioners’ perceptions of the salience of Senge’s five Disciplines in effective work outcomes. International Journal
of Cross Cultural Management, 12(1), pp.102-114.
Coldwell, D.A.L. and Fried, A., 2014. Cross-cultural knowledge management in collaborative academic research. In: A.
Ortenblad, ed. Handbook of research on knowledge management: Adaptation and context. London: Edward Elgar.
pp. 128-156.
Comte, A., 2009. The positive philosophy of Auguste Comte. London: Cambridge University Press.
Darwin, C., 2009. The origin of species and the voyage of the Beagle. London: Random House.

Duffy, P.A., Guertal, E.A., and Muntifering, R.B., 1997. The pleasures and pitfalls of interdisciplinary research in agriculture.
Journal of Agribusiness, [e-journal] 15(2). Available through: Agricultural Economics Association of Georgia,
<www.jab.uga.edu/Library/> [Accessed 11 March 2014].
deKok, B., 2013. Conversation: Interdisciplinarity and innovation. ISRF Bulletin [e-journal] Issue 1. Available at:
<http://www.isrf.org/download/ISRF_BULLETIN_INTRODUCTORY_ISSUE.pdf> [Accessed 15 February 2014].
deKok, B., Hussein, J. and Jeffery, P., 2010. Joining up thinking: Loss in childbearing in resource-poor settings. Social Science
and Medicine, 71, pp.1703-1710.
Easterby-Smith, M. and Malina, D., 1999. Cross-cultural collaborative research: Toward reflexivity. Academy of
Management Journal, 42(1), pp.76-86.
Fuller, S., 2004. I am not a molecule. New Scientist, 2502, 4th June.
Hesse, M.B., 1964. Francis Bacon’s philosophy of science. In: D.J. O’Connor, ed. A critical history of western philosophy.
New York: The Free Press.
Howe, J., 2006. The rise of crowdsourcing. WIRED Magazine, [online] Available at: <http://www.wired.com>
[Accessed 12 January 2014].
Jain, A., Hallensleben, T. and Manger, D., 2013. Reflexivity and innovation: conflicting counterparts? International Journal
of Innovation and Technology Management, 10(6).
King, D., Joosub, T and Coldwell, D.A.L., 2013. ‘Impact of Government Policy on the Market for Corporate Control:
Divestment by South African Conglomerates’, Paper presented at the Strategic Management Society conference,
Strategy and Sustainability, Atlanta, USA. 28 September -1 October.
Lyman, P. and Varian, H.R., 2003. How much information? 2003. [Online] Available at:
<http://www2.sims.berkeley.edu/research/projects/how-much-info-2003> [Accessed 10 March 2014].
Malone, T. W., Laubacher, R. and Dellarocas, C., 2009. Harnessing crowds: Mapping the genome of collective intelligence.
MIT Sloan Research Paper (4732-09).
Merriam-Webster 2016. Positivism. [Online] Available at: <http://www.merriam-webster.com/dictionary/positivism>
[Accessed 20 October 2016).
Nonaka, I. and Hirotaka T., 1995. The knowledge-creating company: How Japanese companies create the dynamics of
innovation, Oxford: Oxford University Press.
Page, S.E., 2008. The difference. How the power of diversity creates better groups, firms schools and societies. Princeton:
Princeton University Press.
Popper, K.R., 1963. Conjectures and refutations: The growth of scientific knowledge. London: Routledge and Kegan Paul.
Popper, K.R., 1992. Unended quest. London: Routledge Classics:
Penny, D., 2009. Charles Darwin as a theoretical biologist in the mechanistic tradition. Trends in Evolutionary Biology, [e-
journal] 1(1). Available at: < http://www.pagepress.org> [Accessed 8 February, 2014].
Pentland, A., 2014. Social physics: How good ideas spread- the lessons from a new science. New York: Penguin Press.
Rosvall, M. and Bergstrom, C.T., 2010. Mapping change in large networks. Plos One [e-journal] 5(1), pp. 1-7. Available at:
<http://www.plosone.org/article> [Accessed 10 March 2014].
Senge, P.M., 1990. The Fifth Discipline: The art and craft of the Learning Organization. New York: Random House.
Spivak, M., 2010. Physics for mathematicians: Mechanics 1. [e-book] Available at: <http://www.Amazon.com> [Accessed 5
June 2015].
Stoyanova, T., 2010. The pros and cons of down sourcing your development work. [online] Available at:
<http://www.cmswire.com/cms/enterprise> [Accessed 19 October 2016].
Thomas, B., 2013. The neuroscience revolution will be crowdsourced. Scientific American, [e-journal] Available at:
<http://blogs.scientificamerican.com/mind-guest-blog/2013/09/11/>
[Accessed 21 January 2014].
Turner, J.H., Beeghley, L. and Powers, C.H., 2012. The emergence of sociological theory. 7th
ed. London: Sage.
Von Hayek, F.A., 1945. The use of knowledge in society. American Economic Review, 35 (4), pp.519-530
The Knowledge Café as a Research Technique
Shawren Singh
School of Computing, University of South Africa, Roodepoort, Gauteng, South Africa
singhs@unisa.ac.za
Abstract: The purpose of this paper is to describe the application of a Knowledge Café as an academic research technique.
Knowledge Cafés are a tool for sharing knowledge among those individuals who participate in the conversations it
encourages. Used in a novel way to non-intrusively listen in on a group of well-experienced e-Government individuals, the
Knowledge Café has provided a technique for critiquing a theoretical conjecture which has lead to its refinement. This
paper describes the steps involved in this refinement process, as well as the how the transcript of the final plenary session
was treated. The previous and the revised theoretical conjectures are provided and the differences addressed. The
application of the Knowledge Cafés technique described here is that the major beneficiaries of this event were not the
participants but the facilitators. However, participants also benefited, but the transcript of the final plenary session of the
Knowledge Café was coded, analysed and used to enhance the theoretical conjecture resulting from the original research.
The critical success factors for the use of a Knowledge Café in this way include finding a group of well-informed
participants, the facilitator/s conducting a well-focused briefing and employing a well-experienced facilitator to manage
the event. In this research, the resulting refined theoretical conjecture is a more satisfactory understanding of how e-
Government operates within its organisational setting.
Keywords: Knowledge Cafés, Theory refinement, Theoretical conjectures, Research Methodology, Hermeneutics
1. Introduction
There are several data gathering approaches that are used in qualitative research (Dey, 2003; Given, 2008).
The research question largely informs the type of data that would be required to answer the research
question. It is not uncommon for qualitative research to use focus groups or interviews. The purpose of this
research is to explore the use of a Knowledge Café in the qualitative research journey. Knowledge Cafés are
widely used in a range of settings where there is a need to encourage individuals to share their knowledge and
experiences. These settings are often organisational where management believes that there is an inadequate
professional discourse about important issues among the organisation’s people. To this end, a Knowledge Café
has been used as a first step in creating communities of interest. However, it is not only organisations who
wish to engage in professional discourse or share knowledge and experience. This type of exchange of ideas,
knowledge and experience is often what is required by academic researchers, especially those following a
qualitative research tradition. It may be argued that the traditional qualitative approaches of employing focus
groups or interviews is an appropriate, well-established method of acquiring data. However, focus groups and
interviews require several iterations. There are several commonalities between focus groups, interviews and a
Knowledge Café. Figure 1, provides a summary of these commonalities.
Figure 1: Different Data Collection Techniques (adapted from Given (2008); Hevner and Chatterjee (2010))
ISSN 1477-7029 29 ©ACPIL
Reference this paper as: Singh S, “The Knowledge Café as a Research Technique” The Electronic Journal of Business
Research Methods Volume 15 Issue 1 2017, (pp29-40) available online at www.ejbrm.com
All of the data collection techniques are driven by a research problem. However, among academic researchers,
it is not well understood how to use the Knowledge Café in their research. Few researchers have used the
Knowledge Café in their research (Brijlall, 2015; Gronau, 2002). In fact, Gurteen (N/D); Lefika and Mearns
(2015); Remenyi (2004) have pointed out that although the Knowledge Café is a useful technique for
researchers, it has largely been ignored in the academic literature. It is the aim of this paper to help fill this
gap.
This paper describes the use of a Knowledge Café (Gurteen, N/D; Lefika & Mearns, 2015; Remenyi, 2004) to
refine a theoretical conjecture developed during a research degree (Singh, 2014) in order to improve the
technologies i.e. e-Government’s utility to a number of its stakeholders.
Having been first used in the 1990s, the Knowledge Café is a well-established technique for sharing knowledge
(Brown & Isaacs, 2005; Gurteen, N/D)within a group of mutually motivated people who have a common
interest about which they are prepared to share their views. It is often used as a learning technique for the
participants, as the exchange of views arising during the Knowledge Café can provide new insights into the
issues being discussed.
The use of the Knowledge Café described here shows how it can both add to the knowledge of the participants
as well as offer interesting insights to researchers in order to refine a theoretical conjecture developed as a
result of their research.
2. Theoretical conjectures
One of the most difficult challenges facing those who create knowledge is the development of new theoretical
insights. Theory development is one of the most creative aspects of academic research and as such it is one
with which researchers often struggle (Handfield & Melnyk, 1998; Shoemaker, Tankard & Lasorsa, 2004).
Moving from data to research findings is often not a major challenge, but, for many individuals, the step
between research findings and a viable theoretical conjecture is often or at least sometimes, problematic.
There are a number of reasons for this, including the fact that academics are sometimes not well acquainted
with the nature of theory and how theory relates to the data or the finding which will have been produced
during the course of the research.
Once a theoretical conjecture has been postulated it is appropriate that it be exposed to critical review in
order to enquire into its verisimilitude and refine it (Kohlbacher, 2005; Remenyi, Money, Price, & Bannister,
2002). Critical reviews are at the heart of academe, and any academic thinking will inevitably benefit from
constructive critical discussion. The required critical review is sometimes obtained by conducting more
interviews or by assembling a focus group. However, as described in this paper, it is also possible to do this
through the use of a Knowledge Café.
3. Research Context
3.1 Setting the Research Scene
South African e-Government applications face several context-specific challenges. Moodley (2005) claims that
exclusive emphasis is being placed on ICT projects, "at the expense of careful analysis and consideration of the
broader economic, social, and political elements that interact to improve the lives of individuals". Moodley
(2005) and Singh (2010) further claim that there are several gaps in policy, gaps in understanding of ICT, gaps
in understanding social process, bureaucratic incompetence, and gaps in understanding "independent human
agency ". In this context, independent human agency refers to people’s spontaneous reactions to events. It is a
challenge to understand these reactions. There are frequent yearly public commitments by successive South
African presidents to provide e-Government services in their state of the nation addresses (State of the Nation
Addresses 2010, 2011, 2012), but these commitments are only partially realized as government is constrained
by limited resources and appropriate skills. It is argued that the government promises e-Government services
because some citizens are engaged in violent, occasionally fatal protests demanding improved service delivery
(Alexander, 2010). These services delivery protests are interpreted as citizens wanting more effective and
efficient service delivery, despite the promises of e-Government. Within these settings, 36 informants within
the South African Public Service Sector who had direct involvement with e-Government policy development or
project implementation and evaluation were interviewed. These were senior state administrators and their
profiles are shown in Table 1.
Shawren Singh
Table 1: The Profile of the Informants

Level Gender Title Total Participants
Male Female Director Specialist Senior Manager Count Total
Senior Manager
National Government 11 5 8 3 5 16 36
Provincial Government 3 0 0 1 2 3
Local Government 3 0 0 1 2 3
Government Agencies 9 5 2 4 8 14
3.2 The research behind the theoretical conjecture

The Knowledge Café discussed here was conducted at the end of a doctoral study into how the delivery of e-
Government applications could be made more effective (Singh, 2014). The study had consisted of the following
steps: After the literature review, a list of key causes of problems in e-Government implementation was
developed. This list was used as the centre pieces of a Delphi Study to verify the importance of the issues. This
was followed by a number of interviews with individuals employed in public services. The resulting transcripts
produced as part of the doctoral research were coded and captured using Atlas-ti. Reports including
conceptual maps were produced and modeled. The original research question was answered and eventually a
theoretical conjecture was presented.
4. What is a Knowledge Café and how can it contribute to research?

A Knowledge Café is traditionally defined as a means of bringing together a group of people who have a
common interest and who will be able to benefit and increase their knowledge by talking together and
listening to each other on the subject of issues related to, and surrounding, that common interest (Gurteen,
N/D; Remenyi, 2004). A Knowledge Café is an effective way of both sharing knowledge, but also of testing
ideas through active conversation.
Talking and reflecting on the concepts represents a type of dialectic experience for those involved. But it is
more than this. A Knowledge Café is also a way in which a researcher can non-intrusively listen to a significant
number of knowledgeable individuals when they are discussing a topic which is central of importance to the
research. It is in this way that the Knowledge Café has been used here.
4.1 The form of the Knowledge Café
There are a number of different formats for operating a Knowledge Café and the particular form a Knowledge
Café is used depends upon the objectives of the facilitator (Brown, & Isaacs, 2005; Gurteen, N/D). Normally a
Knowledge Café will run for about two hours but sometimes an event can be spread over a whole day.
Sometimes a Knowledge Café can result in a formal report which will later be published. Groups participating
in a Knowledge Café will sometimes nominate a chairperson while others will not bother with this degree of
formality. The Knowledge Café described here takes a simple form and is conducted over a two hour period
with an informal reporting and discussion plenary session at the end.
In general, the Knowledge Cafés has several steps (Remenyi, 2004). These steps are outlined in Figure 2.
In the first place, the research question/problem or theory must be developed. Then a significantly sized group
of knowledgeable individuals are assembled who have an interest in that particular issue. This event works
well when the number of participants is 12 or more but less than 40. The Knowledge Café described here
consisted of 28 knowledgeable informants. Fifty informants had been invited to participate and 36 had
accepted the invitation, and 28 arrived on the day.
The first activity is the general briefing which is given to the whole or plenary group and explains the specific
objective of the event and describes how it will be conducted. A thorough briefing is critical, as it sets specific
objectives which will determine the trajectory of the subsequent discussion and, therefore, offer some focus
to the possible ideas which might arise during of the event. If inappropriately briefed, the Knowledge Café can
become a talking shop with little opportunity for a valuable outcome in terms of either critique of what has
been done, or suggestions for future development of the research.

Figure 2: Application of the Knowledge Café

Then the participants are invited to break-up into smaller groups to explore the topic of the Knowledge Café.
The reason why smaller groups are used is that some individuals are nervous about speaking in front of large
numbers of people. Also, in a small group everyone has a better chance of having his or her ideas heard and
discussed. It is not a prerequisite that everyone should speak who participates in a Knowledge Café, but if
anyone remains silent throughout the whole process then it is probable that he or she has lost an interesting
opportunity to air their experience and knowledge.
There is an important philosophical underpinning to the conversations which are held in these small groups
and that it is more important to listen and understand other people’s point of view than to develop a stunning
argument. This helps to ensure that the criticism which arises out of these sessions tends to be reflective and
therefore often constructive.
After a period, usually about 60 minutes, discussion in the small groups, all the participants are invited back
into a plenary session. There are no formal reporting requirements from the small groups but the participants
are asked if they would like to share with the plenary group any new insights they have obtained from their
discussions within the small groups. An opening question to this effect is usually enough to spark a lively
discussion or exchange of ideas with the whole group. Sometimes if a small group has nominated a
chairperson or a spokesperson for their group he or she will speak on the group’s behalf, but often it is simply
a matter of anyone who wants to contribute speaking up.
It is the dialectic processes behind the exchange of ideas that the Knowledge Café seeks to engage, so that the
thinking of the participants may be refined.
5. The use of the Knowledge Café in the research project

5.1 Preparing for the research
Using the Knowledge Café in a different way than the normal described method above meant that the
planning for it had to be detailed and meticulous. It would have been easy for this event to have degenerated
into a discussion group without focus which is not always undesirable when the Knowledge Café is being used
in an organisational setting where the main objective is knowledge sharing.
For the research described here, it was decided to hold a two-hour Knowledge Café which was planned to
attract at least 20 experienced participants. An invitation to the Knowledge Café together with a detailed note
on how an event like this functions was prepared. The Knowledge Café was electronically advertised using four
Shawren Singh
special interest e-mail mail groups, these e-mail mail groups were managed by University of the
Witwatersrand Busines School, University of the Witwatersrand’s Link Centre, University of South Africa’s
School of Computing and the University of South Africa’s Alumni group. The individuals that accepted the
invitation to participate were sent a document explaining that the Chatham House Rule would apply. The
Chatham House Rule is a convention whereby individuals who attend a meeting or a discussion may use the
information obtained during that meeting/discussion, but they should not disclose the name of any individuals
from whom such information was obtained. The Ethics Committee at the university was informed and an
ethics protocol was granted. An appropriate venue with breakaway rooms was found.
The invitations were dispatched two weeks before the event. Seventy-five civil servants and other interested
parties were invited to the Knowledge Café. A total of 36 acceptances to attend were received. Of the 36
people who accepted the invitation, 28 people arrived on the morning to participate in the event. The
participants included executives and senior managers in the public sector as well as representatives of service
providers to government departments, and a small number of academics who had research interests in this
field.
5.2 The briefing of the plenary group
On arrival, the participants were addressed on the nature and purpose of the Knowledge Café and were
invited to sign a letter of consent. The letter of consent included the usual clauses required by the university
ethics committee. The participants were reminded that the Chatham House Rule would apply. The research
which had lead to the theoretical conjecture was briefly described and the theoretical conjecture itself was
explained. The group was informed that the purpose of the Knowledge Café was to expose the research
findings to their critical comments, through the conversations with each other, regarding the verisimilitude of
the theoretical conjecture. As the Chatham House Rule applied they were encouraged to be as open and frank
as possible. This briefing took 15 minutes.
5.3 The theoretical conjecture
For the purposes of this paper, it is assumed that readers will be familiar with the various approaches to theory
development including, reflection, modeling, seeking connections with the literature etc (Bazeley, 2013;
Saunders, Lewis, & Thornhill, 2015). These were all applied in the study described here. The result was that the
following theoretical conjecture concerning how e-Government applications implemented was developed:
e-Government applications are driven by a need to modernise government processes in the

expectation of using scarce resources with improved efficiency. These applications are implemented in
an environment that is not conducive to changes in processes and practices. The pressure to modernise
can lead to non-optimal systems development decisions being made and practices implemented. The
decision to modernise is made with little attention to the appropriate strategies to employ, key
stakeholders who are able to deal with and manage the process, and how the process will be
evaluated. The resulting application automates the department tasks but does not improve their
effectiveness. These automate systems expose gaps in the department processes.
On its own, this theoretical conjecture represented a step forward in understanding the e-Government
phenomenon but it was felt that it would benefit from discussions with a new group of individuals who had
appropriate knowledge and experience of the subject. The researchers had considered using extra interviews
and focus groups but it was considered more interesting to use a Knowledge Café on an experimental basis. It
was realised that this Knowledge Café would not be applied in the normal way that Knowledge Cafés are
usually used.
5.4 The small working groups
The plenary group of 28 was then invited to break-up into smaller groups of between 4 and 6 people and to
discuss the theoretical conjecture. They were allowed to self-select the people with whom they wished to
converse. However, they were advised by the facilitators that they would obtain most from the experience of
the Knowledge Café if they chose to discuss the topic with individuals with whom they were not well
acquainted and this advice they accepted. The small groups so formed were then directed to break-away
rooms. Sixty minutes were allocated for this small group conversation.

At the end of the small group discussion, the plenary group reassembled and a 45 minute discussion ensued.
The timings described here of 60 minutes and 45 minutes are arbitrary but these numbers are considered
practical minimums required for a reasonable discussion.
A Knowledge Café often generates a high degree of animated discussion, and so it was with this event. Once
the plenary group was reassembled the different small groups reported on the issues they had felt more
relevant to the enquiry under discussion. It was clear that there was much to say about this subject.
The facilitators allowed the group to pursue the discussion with minor interventions which mostly involved
seeking the opinions of some individuals who were still relatively shy about speaking out. The facilitators took
notes of the main points raised which were subsequently transcribed for the purpose of analysis into a
transcript.
5.5 The plenary discussion
The 45 minute discussion which was recorded with the permission of the participants resulted in 10 pages of
transcripts which captured the main issues raised by the plenary group. These issues were debated with, in
some cases, a number of different perspectives being expressed. The facilitators, who in this case were the
researcher and his supervisor, remained largely outside of the main discussion which of course meant that the
comments made were not always as focused as perhaps they could have been.
5.6 The transcript of the plenary discussion
A transcript of the main discussion contained a considerable amount of repetition and thus, it was necessary
to summarise it, in order to establish the main learning points from the discussion. The summary of the
transcript is shown in Appendix A.
Appendix A demonstrates that the discussion ranged widely, which evidences the fact that the topic of e-
Government is complex and that there are a number of different perspectives on this topic. This topic touches
on a number of different aspects of management and organisational control and the transcript could be
further summarised in a number of different ways. Given the use of the Knowledge Café as a research tool, the
emphasis here was to look for ideas which could be used in order to facilitate the improvement of the
theoretical conjecture.
To assist the understanding of the transcript it was decided to extract a limited number of key issues from this
summary transcript using a hermeneutic approach (Freeman, 2008). The hermeneutic approach involved
reading the transcript several times and in so doing acquiring an understanding of it as a whole, and of its parts
separately. Thus, the summary was reviewed several times and the issues were thoroughly discussed. Figure 3
contains the 7 issues, with some explanation, which were extracted from the transcript summary.
Key issues extracted from the summary of the conversation among the
plenary group
1. Many of the issues related to e-Government success are management ones. There
does not appear to be a clear understanding of who are the primary custodians of
e-Government systems. Citizens’ interaction needs focused attention. The Public
Service has not embraced a culture of ICT facilitated modernisation and, as a result,
civil servants are often under skilled in terms of ICT potential and implementation.
2. There can be confusion within the public service as to who the real leaders are. Are
Government Minister the real bosses or are civil servants ministers. Maybe both
are? Can a system run with two bosses who may not always see the issues in the
same way? Senior leadership is not accountable for the success or failure of e-
Government systems.
3. e-Governments systems are service delivery systems and should have no party
political dimension.
4. e-Government needs to be managed in terms of a benefit realisation scheme. A
boarder understanding of the potential benefits across most aspects of
government is long overdue. This actually means a major rethink of the technology
required.
Shawren Singh
Key issues extracted from the summary of the conversation among the
plenary group
5. There can be major problems of communications within units of the public services
and knowledge sharing is not often a top priority. There should be a greater focus
on making sure that success is celebrated and that the lessons available from
successful project are known to the organisation.
6. It is important to consider end-user focus groups as a going strategy to engage
with the users of e-Government systems.
7. Stove piping (silos) is still a significant problem within the public sector. Civil
servant should attempt to pool resources and cross-functional systems in order to
improve the efficiency of service delivery.
Figure 3: Keys issues which were identified from the plenary transcript
Reading the theoretical conjecture and the list of key issues it is possible to recognise that there is a degree of
congruency between the original research findings and the opinions of those who participated in the
Knowledge Café. However, it is also relatively clear from both listening to the final plenary session and reading
the transcript of the Knowledge Café that the theoretical conjecture could be improved with the knowledge
obtained from this event.
5.7 Issues to be incorporated in any refinement exercise
The first point which the key issues address is the fact that the success of e-Government is not a matter of the
technology per se, but rather how it is managed, and it emphasises that a modernisation ethos has not been as
fully embraced by the public sector as it is often purported to have been. This is a fundamental issue which
impacts on any change initiative. But in the e-Government environment, this issue is further complicated by
the confusion regarding the ambiguous roles of government ministers vis-à-vis civil servants. Who is actually in
control of the e-Government agenda? On top of this concern was shown for the possibility that e-Government
projects might become influenced by party political considerations.
The issue of benefit realisations, was one which received strong emphasis and of course, this is strongly linked
to the question of communication, which was also raised. The concern for focus groups may also be seen as
connecting to this issue.
5.8 Refining the theoretical conjecture
In the light of these comments obtained from the Knowledge Café and as a result of the further discourse
which they triggered, it was decided to refine the theoretical conjecture and to restate it as follows.
e-Government is particularly sensitive to issues related to management and leadership of which benefit
realisation and communications within and with outsiders are crucial.
Applications driven by a need to modernise government processes in the expectation of using scarce
resources with improved efficiency are important. These applications are implemented in an
environment that is not often conducive to changes in processes and practices due to long stove piping
(silo) arrangements. The pressure to modernise can lead to non-optimal systems development
decisions being made and practices implemented. The decision to modernise is made with little
attention to benefit realisation and how to manage the process, and how the process will be
evaluated.
This is more in line with the opinions expressed during the Knowledge Café. Of course, this new conjecture
cannot in any sense be considered the finished product of the research into e-Government. It is simply another
step forward toward a fuller understanding of how e-Government functions and how it could be improved by
applying the ideas and concepts produced by academic researchers. There is, in fact, no final point in our
understanding of managing organisations as they continue to evolve as fast as they can be researched.
6. Discussion
From the above, it will be seen that a Knowledge Café has a number of similarities to a focus group and
interviews, in that it may be used as an opportunity for the researcher to acquire an understanding of a
situation based on the experiences of a group of knowledge informants. In Table 2, an analysis of focus groups,
interviews and Knowledge Café’s is provided. The following criteria were used in the analysis:

• Basic elements of the approach,

• Selected characteristics,
• Special requirements,
• Management issues, and
• People issues.
This analysis is important because it highlights some of the advantages of using a Knowledge Café. Some of the
advantages of using a Knowledge Café are:
• Basic elements of the approach – A Knowledge Café is a once-off, two-hour scripted event driven by
a facilitator who does not unduly influence the conversation.
• Selected characteristics – A Knowledge Café is flexible enough to explore issues but still protect the
rigor of the approach.
• Special requirements – The bare minimum of equipment is required, most of which the researcher
can carry and setup and the venue does not require specialised equipment.
• Management issues - The researcher can manage the process, equipment, and venue and the cost is
contained as this is a once off event.
• People issues – There is a reduced likelihood of there being a power-trust issue with the informants
and the researcher because the researcher is an observer to the conversations. A further boon to
the informants is that they have the opportunity to network and develop communities of practice
with each other.
It is important to note that a Knowledge Café differs, however, in that the discussion which takes place among
the participants is less directed than it would normally be in a focus group and that by the use of both the
small groups and the plenary sessions a larger number of people can take part in a Knowledge Café. However,
Knowledge Cafés, have not originated from within the research community and have not normally been used
in this way.
Shawren Singh
Table 2: A Comparison of Focus Groups, Interviews and Knowledge Café (Synthesised from Edmunds (2000);
Given (2008); Harper, Jones and Marcus (2013); Hevner, and Chatterjee (2010); Mills, Durepos and Wiebe
(2009); Somekh, and Lewin (2005))

7. Conclusion and summary

Although the application of the Knowledge Cafés as described here has been one of critique it may actually be
used to directly support academic research. It may be incorporated into the structure of a researcher degree
and used at the outset to enhance the researcher’s understanding of the research question or to obtain
suggestions to enhance other aspects of the research.
The success of a Knowledge Café as described here is contingent on:
• Being able to attract a number of appropriately knowledgeable informants to participate in the

event;
• Having access to a suitable facility i.e. a plenary room accompanied by breakaway rooms or a
plenary room that facilitates smaller groups;
• Having an experienced facilitator who is able to manage such an event.
The experimental use of the Knowledge Café described here provided the researchers with an opportunity to
make some material refinements to the original theoretical conjecture. This was expected and was the main
reason for conducting the Knowledge Café. The Knowledge Café provided additional insights which were
necessary to be accommodated in the theory in order to provide it with a more satisfactory interface with
practitioners and allowed it to operate more effectively. In effect these additions to the theory are sine qua
nons for using the theory with the everyday practicalities of e-Government. However, there was no attempt at
any type of hypothesis testing.
The quality of the output of the Knowledge Café is highly dependent on the knowledge and the experience of
the participants. It also depended on the skill of the facilitators and their ability to appropriately brief the
plenary group before the event breaks up into small groups. To be successful the facilitator or facilitators have
to ensure that a thorough transcript of the plenary session is obtained. This has to be carefully summarised
before it can be further used in the research. The analysis of the transcript and or its summary has to be
performed either using content analysis or a hermeneutic approach. Finally, the researcher has to be able to
use the findings to pin point how the theory maybe refined and then perform this task.
None of these are trivial tasks and it requires a considerable level of sophistication with academic research to
be able to accomplish this. But as in the context described above it can enable the research output to be
significantly improved.
References
Alexander, P. (2010). Rebellion of the poor: South Africa's service delivery protests - a preliminary analysis. Review of
African Political Economy, 37(123), 25-40.
Bazeley, P. (2013). Qualitative data analysis: Practical strategies. LA: Sage.
Brijlall, D. (2015). Exploring the Gurteen Knowledge Café approach as an innovative teaching for learning strategy with
first-year engineering students. Groupwork, 24(3), 26-45.
Brown, J., & Isaacs, D. (2005). The world café: Shaping our futures through conversations that matter. CA: Berrett-Koehler
Publishers.
Dey, I. (2003). Qualitative data analysis: A user friendly guide for social scientists: Routledge.
Edmunds, H. (2000). The focus group research handbook: McGraw-Hill.
Freeman, M. (2008). Hermeneutics. In L. M. Given (Ed.), The Sage Encyclopedia of Qualitative Research Methods. The Sage
Encyclopedia of Qualitative Research Methods (pp. 385-388). Thousand Oaks, CA: SAGE Publications, Inc.
Given, L. M. (2008). The Sage encyclopedia of qualitative research methods: Sage Publications.
Gronau, N. (2002). The knowledge café–a knowledge management system and its application to hospitality and tourism.
Journal of Quality Assurance in Hospitality & Tourism, 3(3-4), 75-88.
Gurteen, D. (N/D). Knowledge Cafe. Retrieved 3 October, 2015, from gurteen:
http://www.gurteen.com/gurteen/gurteen.nsf/id/kcafe
Handfield, R. B., & Melnyk, S. A. (1998). The scientific theory-building process: a primer using the case of TQM. Journal of
Operations Management, 16(4), 321-339. doi: http://dx.doi.org/10.1016/S0272-6963(98)00017-5
Harper, C., Jones, N., & Marcus, R. (2013). Research for development: A practical guide: Sage.
Hevner, A., & Chatterjee, S. (2010). Design research in information systems: theory and practice (Vol. 22): Springer Science
& Business Media.
Kohlbacher, F. (2005). The use of qualitative content analysis in case study research. Forum Qualitative
Sozialforschung/Forum: Qualitative Social Research, 7(1). doi: http://nbn-resolving.de/urn:nbn:de:0114-fqs0601211.
Shawren Singh
Lefika, P. T., & Mearns, M. A. (2015). Adding knowledge cafe´s to the repertoire of knowledge sharing techniques.
International Journal of Information Management, 35(1), 26-32. doi: DOI: 10.1016/j.ijinfomgt.2014.09.005.
Mills, A. J., Durepos, G., & Wiebe, E. (2009). Encyclopedia of case study research: Sage Publications.
Moodley, S. (2005). The Promise of E-Development? A Critical Assessment of the State ICT for Poverty Reduction Discourse
in South Africa. Perspectives on Global Development and Technology, 4(1), 1-26.
Remenyi, D. (2004). Knowledge Sharing and Collaboration Knowledge Cafes–Do it Yourself Knowledge Sharing. Paper
presented at the 5th European Conference on Knowledge Management, Conservatoire National des Arts et Metiers.
Remenyi, D., Money, A., Price, D., & Bannister, F. (2002). The Creation of Knowledge Through Case Study Research. Irish
Journal of Management, 23(2), 1.
Saunders, M., Lewis, P., & Thornhill, A. (2015). Research methods for business students (7th ed.): Pearson Education
Shoemaker, P. J., Tankard, J. W., & Lasorsa, D. L. (2004). How to build social science theories. California: Sage publications.
Singh, S. (2010). The South African ‘Information Society’, 1994–2008: Problems with Policy, Legislation, Rhetoric and
Implementation. Journal of Southern African Studies, 36(1), 209-227.
Singh, S. (2014). Towards understanding the process of e-Government application design in South Africa. Reading:
Academic Conferences and Publishing Limited.
Somekh, B., & Lewin, C. (2005). Research methods in the social sciences: Sage.
State of the Nation Addresses (2010). State of the Nation addresses Retrieved 29 September, 2013, from
http://www.info.gov.za/speeches/son/index.html
Appendix A: Summary of the transcript of the plenary group meeting

Success is an ill-defined concept, especially within the public sector. Work needs to be done on
defining what success means before investing in and engaging with technology. If this was done
correctly it could lead to some sort of benefit realisation activity. This would become the main
direction of planning for the use of the technology. The Management of public sector or public sector
investment to ensure the delivery of benefits is often poor. Benefits do not materialise by
themselves. A programme of active benefit realisation would be helpful in achieving the goals set.
This needs to be done by people with a personal commitment to achieving these goals. Benefits do
not arise passively. Although there are some technology issues, the technology is not always as stable
as it is made out to be especially by vendors, and with which there are challenges many of the issues
related to e-Government success are management ones. There can be major problems of
communications within units of the public services and knowledge sharing is not often a top priority.
There should be a greater focus on making sure that success is celebrated and that the lessons
available from a successful project are known to the organisation. If one scans the range of the e-
Government applications which have been successfully implemented to date current systems are
often too focused on the effective collection of revenue from citizens. This approach, although of the
highest priority is in fact quite limiting. A boarder understanding of the potential benefits across most
aspects of government is long overdue. This actually means a major rethink of the technology
required. Is it possible to talk about e-Government without being cognizant of the political process. e-
Government systems are service deliver systems and should have no vested political dimension. But
is this always the case? There are still many issues with regard to prioritising investment in the public
sector. Current e-Government systems are effective at least to some extent in respect of information
dissemination but not at interaction with citizens. There does not appear to be a clear understanding
of who are the primary custodians of e-Government systems. The notion of systems owners which is
sometimes used in the private sector is alien to the public sector. When it comes to the issue of
purchasing systems, there is consultation with the vendor and government. However after the sale of
the system, there is little consultation between the vendor and government. This is a large gap
between the thinking behind the use of e-Government and those who actually use it at the end of the
day. There is inadequate engagement between the citizen and those who conceptualise, plan and
implement e-Government systems. It is important to consider end-user focus groups as a going
strategy to engage with the users of e-Government systems. For citizens to improve the dysfunctional
processes and systems, citizens will have to make their view known. The willingness to complain
needs to be seen within its cultural context. For citizens to accept e-Government systems, citizens
must be literate and computer literate. There are considerable differences of opinion as to how
computer literate the population is actually. There is a level of despondency within the civil service
resulting in civil servant not taking an active interest in ensuring that the systems and process

function optimally. The Public Service has not embraced a culture of ICT facilitated modernisation
and, as a result, civil servants are often under skilled in terms of ICT potential and implementation.
There can be confusion within the public service as to who the real leaders are. Are Government
Minister the real bosses or are civil servants. Maybe both are? Can a system run with two bosses who
may not always see the issues in the same way? Senior leadership is not accountable for the success
or failure of e-Government systems. Stove piping (silos) is still a significant problem within the public
sector. Civil servant should attempt to pool resources and cross-functional systems. Reactionary
change propels the development of some e-Government systems. The public sector is actually many
different worlds, some of which have little contact with one another. Due to this fragmented nature
of government, there has not been a consistent approach to business cases for e-Government. e-
Government systems should be functionally separated from the political system/drivers. For e-
Government systems to be effective, it is required that there is a stable bureaucracy present within
government.
Multiple Imputation by Chained Equations in Praxis: Guidelines and
Review
Jesper N. Wulff1 and Linda Ejlskov2

1
Aarhus University, Department of Economics and Business Economics, Denmark
2
Aalborg University, Department of Health, Science and Technology, Denmark
Abstract: Multiple imputation by chained equations (MICE) is an effective tool to handle missing data - an almost
unavoidable problem in quantitative data analysis. However, despite the empirical and theoretical evidence supporting the
use of MICE, researchers in the social sciences often resort to inferior approaches unnecessarily risking erroneous results.
The complexity of the decision process when encountering missing data may be what is discouraging potential users from
adopting the appropriate technique. In this article, we develop straightforward step-by-step graphical guidelines on how to
handle missing data based on a comprehensive literature review. It is our hope that these guidelines can help improve
current standards of handling missing data. The guidelines incorporate recent innovations on how to handle missing data
such as random forests and predictive mean matching. Thus, the data analysts who already actively apply MICE may use it
to review some of the newest developments. We demonstrate how the guidelines can be used in praxis using the statistical
program R and data from the European Social Survey. We demonstrate central decisions such as variable selection and
number of imputations as well as how to handle typical challenges such as skewed distributions and data transformations.
These guidelines will enable a social science researcher to go through the process of handling missing data while adhering
to the newest developments in the field.
Keywords: Multiple imputation by chained equations, MICE, missing data, guidelines, review, R
1. Introduction
Valid tools to handle missing data should be in every data analyst’s toolkit. Not having these tools force data
analysts to resort to inferior default approaches when analysing incomplete data unnecessarily risking
erroneous results. The most often used technique to cope with incomplete data is to apply some form of
complete case analysis as e.g. listwise deletion or mean-value imputation. In the case of the former, a
respondent is left out if data are missing on one or more of the variables included in the analysis, while the
latter approach fills in the missing values with the mean calculated from the observed data. To remove
respondents that lack data on certain variables or single-impute missing observations can be harmless, but
often it leads to biased estimates and incorrect standard errors (Sterne et al. 2009). Thus, it is not surprising
that research methodologists only recommend to use list- / pairwise deletion or single imputation techniques
in rare cases (Donders et al. 2006; Greenland & Finkle 1995; King et al. 2001; Newman 2014).
Multiple imputation (MI) is an attractive alternative to help researchers make the best use of their data. While
MI is a powerful technique, unleashing its full potential requires its user to make a series of important
decisions. If the wrong choices are made, it may compromise the results. This dire potential consequence
coupled with the complexity of the decision process may discourage potential users from adopting the
technique. In part, it may explain why MI is rarely used in practice in psychology (Roth 1994), political science
(King et al. 2001) and management research (Newman 2014), which are fields that often rely on datasets
containing missing values. This is unfortunate as very convincing evidence exists suggesting that MI provides
more precise and reliable results than simpler methods (Graham & Schafer 1999; Little & Rubin 1987; Rubin &
Schenker 1986; Rubin 1996).
We acknowledge that for a researcher unfamiliar with MI getting started may seem like a daunting task. Thus,
in this paper, we first offer a non-technical overview of the state of the art and develop concrete guidelines on
how to implement MI-techniques with a focus on multiple imputation by chained equations (MICE) in section
2. Our goal is to spread the use of MI-techniques and thereby increase the quality of the research produced. In
addition, data analysts who already implement MI-techniques may also benefit from this paper by updating
their knowledge with the newest developments in the field as e.g. predictive mean matching, random forest
imputation and model selection.
ISSN 1477-7029 41 ©ACPIL
Reference this paper as: Wulff J and Ejlskov L, “Multiple Imputation by Chained Equations in Praxis: Guidelines and Review”
The Electronic Journal of Business Research Methods Volume 15 Issue 1 2017, (pp41-56) available online at
www.ejbrm.com
In section 3 we demonstrate how to implement the guidelines in praxis and overcome common challenges
facing the researcher when using MICE to handle missing data. We demonstrate how to handle missing data
using MICE and compare the results of different imputation methods. The demonstration is based on data
from the European Social Survey using R. R is a free software environment for statistical computing enabling
the reader of this text to apply the presented techniques without depending on access to a specific type of
commercial software (R Core Team 2015). Further, using R we can demonstrate some of the newest
developments in the field not yet accessible through other standard software packages. The R-syntax is
available in the appendix. https://www.researchgate.net/publication/316190594_Appendix_with_R_code
2. Multiple imputation
Before describing MI, it makes sense to briefly describe the mechanisms that makes the data missing: Missing
completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). MCAR is when
the probability of the data missing depends on neither the observed nor the unobserved data. Missing data
are MAR when the probability of missing data to some extent depends on the observed data. Finally, data are
MNAR when the probability of missing data depends on the missing data values themselves. Missing data are
almost never MCAR, but instead somewhere in between MAR and MNAR (Graham 2009). If the missing data
are tending towards the MAR category, MI-techniques outperform standard techniques such as listwise
deletion (Rubin 1996).
MI was proposed by Rubin (1987) as a statistical technique for handling missing data. At its core, MI uses the
distribution of the observed data to estimate a set of likely values of the data that are missing. MI estimates
these values M times each time incorporating a random component to reflect the uncertainty about the
missing values. After running the procedure, we are left with M different datasets on which we perform the
desired analysis (e.g. linear regression). Next, we take the average of the parameter estimates across the M
datasets resulting in one unbiased parameter estimate for each parameter in our model. The standard errors
are calculated using Rubin’s (1987) formula. An illustration of the whole MICE-process based on Rubin (Rubin
1987) is shown in Figure 1 below.
Figure 1: The MICE-process (Rubin 1987)

The power of MI lies in its many imputations. While every single imputation is imprecise, the combination of
several imputations takes the uncertainty of each imputation into account. For MAR or MCAR data the pooled
parameter estimates are unbiased and standard errors are corrected appropriately (King et al. 2001). In other
words, traditional hypothesis testing based on MI-standard errors are more accurate (Newman 2014).
2.1 MICE
In praxis, we often encounter missing values in multiple variables. MICE is a popular adaption of MI and is
available for the user through the most commonly used software packages. MICE changes the imputation
problem to a series of estimations where each variable takes its turn in being regressed on the other variables.
MICE loops through the variables predicting each variable dependent on the others. This procedure provides
great flexibility as each variable may be assigned a suitable distribution, e.g. poisson, linear or binomial (White
et al. 2011).
MICE runs through an iterative process: In the first iteration, the imputation model for the variable with the
least missing values is estimated using only complete data. Next, the variable with the second least missing
Jesper N. Wulff and Linda Ejlskov
values is imputed using the complete data and the imputed values from the last iteration. After each variable
has been through this process, the cycle is repeated using the data from the last iteration. Typically, ten
iterations are performed where the imputed values after the 10th and final iteration constitutes one imputed
data set (Stuart et al. 2009).
As mentioned above, MICE has an important ability to handle different variable types as each variable is
imputed using its own imputation model (Bartlett et al. 2014). This provides flexibility and makes it possible to
impute data sets that include hundreds of variables (He et al. 2010). For instance, continuous variables may be
modeled through linear regression, binary variables through logistic regression etc. (Chevret et al. 2015). One
downside of MICE is that it does not have the same theoretical justification as e.g. multivariate normal
imputation. In praxis, this does not seem to be an issue (White et al. 2011). Another drawback is the challenge
of choosing an appropriate imputation model. Later in this paper, we walk through an example of building an
imputation model.
Random forest imputation presents a recently developed alternative to standard MICE procedures. Random
forests are an extension of classification and regression trees. Classification and regression trees use a binary
splitting approach recursively subdividing the data based on the values of the predictor variables. Random
forests build many trees each time varying the sample and the predictors. Consequently, a new bootstrapped
sample of observations and predictors are selected for each tree (Hastie et al. 2009).
Under certain circumstances, random forest imputation may offer an attractive alternative to standard MICE.
First, in standard MICE omitting important interactions and other non-linear terms may lead to biased results
(Seaman et al. 2012). Random forest imputation, on the other hand, is a non-parametric technique well-suited
for handling complex non-linear relationships. This is especially beneficial when the imputation model is
difficult to specify. Simulation with data sets including interactions have demonstrated that random forests
MICE (Shah et al. 2014; Doove et al. 2014) and even MICE with regular regression trees (Burgette & Reiter
2010; Doove et al. 2014) result in less biased parameter estimates than conventional standard MICE.
Random forest imputation is also an attractive alternative in other cases where standard MICE is simply not
feasible. Without prior information, standard MICE is not possible in high-dimensional settings when the
number of predictor variables exceed the number of observations (Hardt et al. 2012; Zhao & Long 2016) and
may not converge if highly correlated predictor variables are included in the imputation model. Because
random forest imputation still needs to be tested on a larger range of data, standard MICE may still be
preferred in standard cases. However, due to promising results, it is an option in situations where standard
MICE is known to produce biased estimates (e.g. when facing severe non-linarites) or is simply unfeasible (e.g.
when dealing with high-dimensional data).
2.2 The amount of missing data
Before proceeding with the example, it is relevant to briefly discuss what the partial missing data rate means
for whether one should consider MI-techniques. Newman (2014) suggests a rule-of-thumb of using MI when
confronting a missing-rate of 10% of higher. At this level, the difference between MI and simpler techniques is
expected to be substantial. Multiple studies have investigated the impact of different missing rates, e.g. from
10% - 90% (Janssen et al. 2010) and 2.5% - 30% (Knol et al. 2010), and found that MI outperforms listwise
deletion and similar techniques under different missing mechanisms and sample sizes. For instance, it is
possible for an odds-ratio estimate to deviate systematically from the true value even for at 5% MAR-missing
rate (Knol et al. 2010).
A natural question is whether the missing rate can be too high to use MI. Studies have shown that MI is
unbiased to around 50%, but gets unstable for higher rates, especially if the data have skewed distributions
(Haji-Maghsoudi et al. 2013; Lee & Carlin 2012). However, this does not imply that listwise deletion should be
preferred as MI exhibits superior performance even for a 75% data loss despite biased estimates (Marshall et
al. 2010). In praxis, though, high computation times caused by a very high missing data rate may make MI
infeasible.
MI-scholars have investigated and debated whether listwise deletion is ever appropriate. The short answer
seems to be only in rare cases (Heitjan & Rubin 1990; King et al. 2001). As pointed out by Newman (2014), the
10% rule-of-thumb illustrates an important point despite the arbitrary threshold: When the amount of missing

data is small it will most likely make a very little difference to use MI (Marshall et al. 2010). In summary,
research indicates that MI is the superior choice when facing missing rates above 10% and in some cases
already around 5%. To sum up the suggestions in the literature, we have developed the following guidelines
illustrated in the decision tree in Figure 2. In some cases it might be worth considering MI even though the
partial missing rates are below 10%. For example, in cases when including a number of variables in a
regression model with low partial missing rates. In such a case this might result in a total missing rate in the full
regression model that is substantially higher than in a simple bivariate regression model. Thus, we suggest
always to check for both total missing rates as well as partial missing rates.
Figure 2: MICE decision tree
3. Multiple imputation: An example

In the following, we walk through an example of using multiple imputation on a real data set following the
guidelines we have presented in Figure 2. In our example, we are not going to explicitly choose between the
methods but instead compare them in terms of runtime and end results. However, for each method we do
detail its appropriateness in relation to the real-life example. As we explain below, the first phase in the
specification part in step 2 demands more from the analyst when using standard MICE than when using
random forest MICE. The analysis procedure of the imputed data, however, does not differ depending on the
chosen method of imputation.
3.1 Data
The real life data used in the example stem from The European Social Survey (ESS). The ESS monitors and
charts a long range of individual attitudes, beliefs and behaviour patterns in Europe and is a cross-national
repeated cross-sectional survey with the first round occurring in 2002 and has since then been repeated every
second year (ESS Round 6: European Social Survey Round 6 Data 2012). In this paper, we have used the 6th
round of ESS, which in addition to the core modules has a special emphasis on personal and social wellbeing.
The target population for the survey consists of persons aged 15 and over that are resident within private
households and in total 54,673 participants from 29 surveyed countries answered the interview-administered
questionnaire. For this example, we are interested in investigating how different attitudes towards
immigration are associated with happiness among the elderly population in Europe controlled for a range of
covariates. After excluding people below the age of 71, 7,582 participants were included in the shown
example.
3.2 Step 1: Data evaluation
The sample is summarized in Table I. As it can be seen in Table I, the partial response rate is above 10% for one
of the variables and above 5% for five variables. This means that using a MICE framework to handle missing
data is appropriate. The next phase is choosing the appropriate imputation method. This choice is discussed
throughout the remainder of section three.
Table I: Summary of subsample
Number Proportion
Names Description Range Mean(sd)
missing missing
Analysis model variables

happy Level of happiness 102 1.35 1-11 7.85(2.32)
Feel appreciated by people you
flapppl 140 1.85 1-11 8.86(1.91)
are close to
agea Age 133 1.75 71-103 77.54(5.29)
gndr Gender 3 0.04 1-2 1.58(0.49)
health Self-rated health 17 0.22 1-5 2.9(0.95)
hinctnta Household's total net income 1372 18.1 1-10 3.54(2.26)
Government seeks to reduce
grdfincc 550 7.25 1-11 5(2.89)
income differences
eduyrs Education in years 127 1.68 0-42 9.97(4.51)
Cultural life undermined or
imueclt 701 9.25 1-11 6.03(2.54)
enriched by immigrants
Immigrants make country worse
imwbcnt 689 9.09 1-11 5.52(2.39)
or better place to live
Immigration bad or good for
imbgeco 658 8.68 1-11 5.52(2.52)
country's economy
cntry Country of residence
Auxilliary variables
fltlnl Loneliness 77 1.02 1-4 1.69(0.9)
Receive help and support from
rehlppl 112 1.48 1-7 5.99(1.33)
people you are close to
Provide help and support to
prhlppl 149 1.97 1-7 5.96(1.39)
people you are close to
How many people with whom
inprdsc you can discuss intimate and 185 2.44 1-7 3.41(1.54)
personal matters

Number Proportion
Names Description Range Mean(sd)
missing missing
Take part in social activities

sclact 349 4.6 1-5 2.54(1.09)
compared to others of same age
hincfel Feelings about income 75 0.99 1-4 2.27(0.95)
dweight Design weights 0 0 0.03-4 0.87(0.38)
pweight Population weights 0 0 0.03-5.5 1.06(1.3)
3.3 Step 2: Model specification

Below, we discuss selected considerations and choices that are central in the specification of the imputation
model when using MICE. As it will become evident, this step is often the most cumbersome. We provide
examples under each point using the data described above and the R-package mice (van Buuren & Groothuis-
Oudshoorn 2011).
3.3.1 Choosing variables
MICE needs an imputation model and an analysis model. While the imputation model is used for filling in the
missing values, the purpose of the analysis model is to analyse the imputed data afterwards. These models
need to be compatible. This means that every relationship included in the analysis model needs to be included
in the imputation model regardless of whether they contain missing values or not (von Hippel 2009). Note that
this includes all the variables in the analysis model including the dependent variable (Graham 2009). If the
intent is to use more than one analysis model, the imputation model needs to include all the variables included
in all of the analysis models (White et al. 2011). Variables that are mutually exclusive should be included as one
combined categorical variable to avoid that a respondent may belong to several categories simultaneously. If
the analyst is conducting survival analyses, the censoring variable and a variable containing the Nelson-Aalen
estimate of cumulative hazard function should be included as well (White & Royston 2009). If the variable
selection makes the imputation infeasible due to too many variables relative to the observations or unstable
due to highly correlated predictors, random forest imputation may be considered (Shah et al. 2014).
Our analysis models in the current example include the following variables (also see Table I): Level of
happiness (outcome variable), feeling appreciated by the people close to you, age, gender, self-rated health,
income, government actions to reduce income differences, years of education, three questions about opinions
on immigration and country of residence.
3.3.2 Interactions and other transformations
If the analysis model contains interactions and the imputation is performed through standard MICE, the
analyst must specify these in the imputation model as well to avoid bias in the analysis (Seaman et al. 2012).
This principle holds for other nonlinear terms, e.g. log-transforms or polynomials, and may be summed up as
transform-then-combine (Enders et al. 2014; von Hippel 2009). All other methods involving correcting values
after the imputation are not advised (von Hippel 2009). For instance, it might be tempting to adjust the
imputed values for a squared variable so they correspond exactly to the actual squared values of the imputed
variable. Even passively generating interaction terms as it is possible in some statistical packages may lead to
biased estimates (von Hippel 2009). If the data analyst suspects conceptually or empirically that important
nonlinear terms are left out, random forest imputation may offer an alternative where the researcher does not
specify the interactions in the imputation model (Stekhoven & Bühlmann 2012).
To illustrate the principle of incorporating non-linear terms, we include the interaction between the two
income-related variables (government seeks to reduce income differences (grdfincc) and the household's total
net income (hinctnta)) into the standard MICE model. To reduce collinearity, we mean-center the variables
before the generating the interaction term (von Hippel 2009).
3.3.3 Composite scales
When composite scales generated from other variables are in the analysis model, the researcher may choose
to form the scale before or after the imputation. Graham (2009) suggests to generate the scale before the
imputation when at least half of the items are observed, the items exhibit high coefficient alphas and have
high item-total correlations. In other cases, the recommendation is to first impute and then construct the
scales. Even though both approaches results in unbiased estimates, the most efficient results is generally
achieved by imputing first and then combining the items (Gottschall et al. 2011).
After our imputation, we generate a scale that will reflect the attitude towards immigration. We will do this by
summing three immigration-related variables: The degree you believe cultural life is enriched by immigrants;
immigrants make your country a better place to live; immigration is good for the economy. A higher score on
the index reflects a more positive attitude towards immigration.
3.3.4 Auxiliary variables
Auxiliary variables are not used in the analysis model but are included in the imputation model as they are
suspected to contain information about the missing values. Including auxiliary variables that are correlated
with the analysis variables or are good predictors of the missing values reduces bias and make the MAR-
assumption more reliable (Schafer 2003). Studies have shown that a more loose strategy where extra variables
are added on a looser basis is preferred over a more restrictive strategy as the marginal costs of adding
variables is often low (Collins et al. 2001). Still, the ratio of variables to cases with complete data should not fall
below 1:3 to avoid downward bias in regression coefficients and precision decrease (Hardt et al. 2012).
Variables that are highly correlated with an incomplete variable should be included in the imputation model
(Little 1995; Graham 2009). Finally, if the data contain survey weights these should be included as a covariate
in the imputation model (Kim et al. 2006).
The included auxiliary variables can be seen in Table I. We follow a relatively loose strategy but only include
one variable in the imputation of another if it has a correlation of at least 0.2 to shorten the computation time.
The ESS provide survey weights why these are also included in the imputation model1.
3.3.5 Handling skewed and non-normal variables
As mentioned above, the rule for choosing an imputation model for a given variable in MICE is to choose the
regression model that would also have been appropriate for ordinary analyses. For instance, linear regression
for continuous variables, logistic (logit) for binary, poisson for cardinal variables etc. Before the imputation, it
is advantageous to run a model with each variable as a dependent variable and check the model’s
assumptions. If the distribution of a variable is non-normal and the imputation model assumes normality, the
distribution of the imputed values may not match the distribution of the observed values well (Morris et al.
2014). However, simulations have shown that a normal imputation model with non-normal data works
surprisingly well (Graham & Schafer 1999). Besides turning to random forest imputation, predictive mean
matching offers an alternative imputation modeling approach through MICE. This approach is described below.
3.3.6 Predictive mean matching
Predictive mean matching (PMM) is an increasingly popular tool in the MICE-toolbox. PMM produces imputed
values that resemble the observed values better than methods based on the normal distribution (White et al.
2011). If the original variable is right-skewed, PMM will produce imputed values that follow the same
distributional pattern. The reason for this is that PMM uses the predicted value for a given observation to
identify similar observations. Using the identified observations, a matching set is created containing k matches.
Next, PMM draws from this set at random. Consequently, PMM uses the real values from individuals with real
data. This prevents unwanted extrapolation beyond the range of the data (Little 1988).
The analyst can specify the size of the matching set, i.e. how many similar cases a set should contain. The
default k=1 in Stata and SPSS has been shown to lead to estimated standard errors that are too low resulting in
t-statistics that are too large (Morris et al. 2014). Some research has found a small advantage of k=3 over k=10
(Schenker & Taylor 1996) while the literature contains recommendations of k=10 (Morris et al. 2014). A large k
is likely to be more effective in larger samples and may have poor performance in small samples as the most
similar observations may become too different.
In praxis, PMM has shown performance close to correctly specified parametric models and better than poorly
specified parametric models characterized by non-normality (Morris et al. 2014; Schenker & Taylor 1996) and
moderate skewness (Marshall et al. 2010) despite that the method lacks a formal mathematical justification
1
Survey sampling weighting adds another layer of complexity to multiple imputation (Azur et al. 2011). Our sample code
contains an example of how one might combine multiple imputation and survey weights. However, the reader should be
aware that this is an area of on-going research where few reliable guidelines exists. For more details on the matter, we
refer to Kim et al. (2006) and Schenker et al. (2006).

(Kenward & Carpenter 2007). Especially the speed of PMM seems to provide it with an advantage for large
datasets, but may run into problems for small samples due to the matching set issue.
3.3.7 Random forests
If the decision falls on random forest imputation, there is little to specify for the analyst besides the number of
trees. The recommended number is 10 (Shah et al. 2014), which is also the default in the mice package. Similar
to standard MICE, it is preferable to include auxiliary variables in the imputation. As mentioned above, the
researcher does not compute interaction and higher-order terms beforehand. It remains to be investigated
whether other transformations such as log-transformations follow the transform-then-combine rule-of-thumb
in the case of random forest. Situations when random forest imputation may be preferred are discussed above
and an overview of the decision process is available in Figure 2. Below, we compare results from MICE (using
software defaults), MICE with some variables imputed through PMM, MICE using full PMM and MICE using
random forests.
3.3.8 Comparing methods
In Table II below, we show four different imputation approaches. For MICE Default, we let the software choose
the imputation method for each variable. For numeric variables like agea, it chooses PMM, while factor
variables with more than two levels are imputed using multinomial logistic regression (polyreg) and two-level
factors are imputed using logistic regression (logreg). For MICE Mixed, we have coded factor variables with
more than 10 levels as numeric, thus letting them be imputed by PMM. For MICE Full PMM, we impute every
variable using PMM, while MICE RF imputed every variable using random forest imputation.
In Figure 3, we graphically compare the distributions of the observed (blue) and imputed (red) composite
immigration variable (immi) and one of the immigration-related variables (imueclt) across imputation models.
The distributions are very similar even though MICE Default uses multinomial logit while MICE mixed and Full
PMM use PMM (Table II). Random forest seems to deviate a little more from the observed data. The close
similarities are also apparent when observing one of the variables used to generate the composite variable.
Figure 3: Distributions of observed and imputed values
3.4 Running the imputation model

Before the analysis of the imputed data, we briefly discuss some practicalities considering running the
imputation model. The subjects include the number of imputations, model convergence and computation
time.
3.4.1 Number of imputations
The data analyst specifies the number of imputations. Earlier work on MI has focused on efficiency and
recommends that 3 or 5 imputations are sufficient (Schafer 1997) and that 10 are more than enough (Fichman
& Cummings 2003; Schafer 1999). Recent research has turned its focus from efficiency to statistical power
instead recommending that the number of imputations should be at least equal to the percent of missing data,
i.e. 20% missing data requires 20 imputations (Graham et al. 2007; Bodner 2008; White et al. 2011). This rule-
of-thumb minimizes the loss in statistical power in most situations (White et al. 2011).
Table II: Overview of imputation methods for different variables
Variable MICE Default MICE Mixed MICE Full PMM MICE RF
Analysis model variables
happy polyreg pmm pmm rf
flapppl polyreg pmm pmm rf
agea pmm pmm pmm rf
gndr logreg logreg pmm rf
health polyreg polyreg pmm rf
hinctnta pmm pmm pmm rf
grdfincc pmm pmm pmm rf
eduyrs pmm pmm pmm rf
imueclt polyreg pmm pmm rf
imwbcnt polyreg pmm pmm rf
imbgeco polyreg pmm pmm rf
cntry
income_int pmm pmm pmm
Auxilliary variables
fltlnl polyreg polyreg pmm rf
rehlppl polyreg polyreg pmm rf
prhlppl polyreg polyreg pmm rf
inprdsc polyreg polyreg pmm rf
sclact polyreg polyreg pmm rf
hincfel polyreg polyreg pmm rf
dweight
pweight
Imputation time
Minutes 256.6 71.9 6.7 64.9
Notes: The blank cells indicate that the variable was not imputed either due to non-missingness (cntr, dweight, pweight) or
because the variable was not included in the imputation model (income_int for MICE RF). Computation time is based on an
average of three runs on an Intel® Core™ i7-5600U CPU 2.60 GHz with 16 GM RAM running 64-bit Windows 7 Enterprise
and R version 3.2.4 (2016-03-10).
In our data example, the highest single-item missing rate is 18.1%. In addition, we have relatively high missing
rates on the immigration variables (see Table I). Following the rule-of-thumb above, we run 20 imputations.
3.4.2 Model convergence
As explained earlier, MI usually runs 10 iterations with the 10th iteration constituting one imputed dataset. The
first 9 iterations are called the burn-in period and are usually sufficient for the process to stabilize (Stuart et al.
2009). Model convergence can be examined by plotting the mean and standard deviation (sd) as a function of
the iterations for each variable. In Figure 4 below, we assess model convergence for three variables by plotting
the mean and variance of the imputations against the iteration number, respectively. That is, each colored line
represents an imputation and for each imputation we plot the mean (left side of figure 4) and sd (right side of
figure 4) for each of the 10 imputation iterations. Convergence should be assessed for each variable in the
imputation. If a clear trend emerges, the researcher should increase the burn-in period. Below, we show

iteration plots for the mean and standard deviation of our interaction term variable and two of the
immigration variables (MICE Default).
From Figure 4, we learn that in our case the means and standard deviations stabilize rather quickly. The
interaction term (income_int) shows the most exemplary convergence while the two immigration items take
around five iterations to stabilize. Because we know that the two items are highly correlated, this is to be
expected. The user can always increase the burn-in period to observe if the chains indeed stabilize. With little
reason to increase the burn-in period in our case, we move on to the analysis of the results after a brief note
on computation time.
Figure 4: Convergence plots
3.4.3 Computation time

Even though the distributions presented in Figure 3 are very similar, the difference in time it took to impute
the data is not: While the MICE Default took 256.6 minutes to run, it took only a about fifth as long for MICE
Mixed and just a total of 6.7 minutes for the full PMM (Table II). This illustrates an important point about the
enormous speed advantage PMM can have over competing methods: As imputation models grow increasingly
complex or in cases of large amounts of missing data, PMM becomes ever more attractive. If the imputation
process becomes unreasonably long, factor variables with several levels become obvious candidates for
reducing imputation time through PMM. In our case, random forest completes almost as fast as MICE mixed,
but still not nearly as fast as full PMM. This illustrates that random forest imputation may be viable alternative
in cases where lowering the computation becomes valuable, but a full PMM solution for some reason is
undesirable.
3.5 Step 3: Analysis of imputed data
After the imputation, we have m complete datasets. We estimate the model (e.g. linear regression) on each of
the m datasets and combine the estimates to one combined result. Several steps of analysing these datasets is
often very similar to running a regression on a single dataset as the process is automated in modern statistical
software. This combination step is the same regardless of the imputation procedure selected above. As we
discuss below, some issues as e.g. variable selection become quite complicated when using MI.
Below in Table III, we display the estimated coefficients for the analysis model while comparing our four
procedures used to impute the missing data as we described above. In addition, we compare the results to a
complete case analysis where the missing data are removed using listwise deletion. The complete case results
deviate notably from the imputed results. The effect size of household income (hinctnta) is approximately 11%
larger when not considering the missing data. It is also noteworthy that the interaction term is smaller when
running the regression on the imputed data compared to the complete data. When conducting traditional
hypothesis testing, the coefficient on the complete data would be significant at an alpha of 1% (t = -2.78, p =
0.005), while it would only be significant at an alpha of 5% on the imputed data (t = -2.09, p = 0.037). Thus,
results may appear more convincing than they really should be when not considering the uncertainty of the
missing data.
The results from the default method and the full PMM are very similar. Recall from Table II, however, that the
default method took almost 40 times longer to run because of the much more computationally heavy
multinomial logit. This repeats the point about the PMM being an attractive alternative when the imputation
process becomes unreasonably long.
As the interpretation of the coefficients and standard errors are exactly as in regular regression, we will not
dwell further on these. Instead, we move on to areas where some subtle differences to analysing non-imputed
data exist.
Table III: Comparing linear regression results from MICE and complete case
Complete MICE Full
Variable MICE Default MICE Mixed MICE RF
case PMM
Intercept 3.869 (0.487) 3.692 (0.393) 3.700 (0.389) 3.662 (0.389) 3.654 (0.389)
flapppl 0.309 (0.014) 0.293 (0.012) 0.296 (0.012) 0.297 (0.012) 0.287 (0.012)
agea 0.013 (0.005) 0.012 (0.004) 0.011 (0.004) 0.012 (0.004) 0.012 (0.004)
gndr -0.044 (0.053) -0.012 (0.045) -0.011 (0.045) -0.015 (0.045) -0.020 (0.045)
health2 -0.379 (0.107) -0.348 (0.092) -0.341 (0.092) -0.343 (0.092) -0.347 (0.092)
health3 -0.777 (0.106) -0.819 (0.091) -0.815 (0.091) -0.816 (0.091) -0.828 (0.091)
health4 -1.492 (0.122) -1.573 (0.102) -1.571 (0.102) -1.571 (0.102) -1.572 (0.102)
health5 -2.284 (0.168) -2.271 (0.138) -2.258 (0.135) -2.269 (0.135) -2.271 (0.135)
eduyrs -0.010 (0.007) -0.005 (0.006) -0.005 (0.006) -0.006 (0.006) -0.003 (0.006)
hinctnta 0.098 (0.013) 0.088 (0.011) 0.087 (0.011) 0.087 (0.011) 0.084 (0.011)
grdfincc 0.071 (0.01) 0.076 (0.009) 0.076 (0.009) 0.078 (0.009) 0.068 (0.009)
income_int -0.011 (0.004) -0.008 (0.004) -0.008 (0.004) -0.008 (0.004) -0.009 (0.004)
immi 0.030 (0.004) 0.033 (0.004) 0.032 (0.004) 0.033 (0.004) 0.034 (0.004)
Notes: Standard errors in parentheses. Country fixed effects are included in all models.
3.5.1 Fit statistics
Statistics that are not estimators cannot be combined using Rubin’s (1987) rules. White et al. (2011) provide an
overview of the most common statistics and whether they may be combined, transformed or should not be
combined. For linear models, the mean of adjusted R2 across imputed datasets may be supplemented by
information about the percentiles. Fischer’s r to z transformation has been suggested in the literature (Harel
2009), but if it is not substantially different, the simple mean may be preferred. For likelihood-based models,
as e.g. logistic regression, the likelihood ratio is popular when analysing regular non-imputed data. A method
for calculating the likelihood ratio has been suggested (Meng & Rubin 1992), but is has been shown to be less
accurate than the easily obtained Wald test statistic (Wood et al. 2008).
Here, we focus on the results from the default MICE method. When using Fischer’s r to z transformation, the
adjusted R2 for our model is 0.382 (low 95: 0.364; high 95: 0.399). In our case, there is not much variability in
the adjusted R2 across the 20 imputed datasets. If we compute the adjusted R2 for a model excluding the
interaction term, this simpler model would even fit a little better (Adj. R2: 0.385; low 95: 0.368; high 95: 0.403).
We can compute a Wald test statistic and compare the model with to the model without the interaction term.
This results in a test statistic of 4.386 with an associated p-value of 0.037. Using an alpha of 5%, traditional
statistical inference suggests that the H0 of no difference between the models should be rejected. However,
these model comparisons consider neither model uncertainty nor do they penalize model complexity
adequately. To do this, we move on to a brief discussion of model selection and averaging in the context of
imputed data.

3.5.2 Model selection and model averaging

Model selection considers choosing the single best model among a set of candidate models. This preferred
model would often be the one that minimizes the generalization error commonly approximated through cross-
validation or information-based criteria as e.g. Akaike’s Information Criterion (AIC). Model averaging deals with
the uncertainty about the model selection process and acknowledges that there may be many good models
that describe the data.
Schomaker and Heumann (2014) suggested and tested different ways of integrating model selection and
averaging approaches into the MI process. The different algorithms are made available through the MAMI-
package (Schomaker 2015). The package allows for several combinations of selection criteria for both model
selection and averaging. Below, we show results from their stepwise variable selection based on AIC (Posada &
Buckley 2004). This procedure performs stepwise variable selection on each imputed data set. If a given
variable is selected its estimate is different from zero while it is zero if the variable is not selected. Thus,
coefficients of variables selected in only a few imputed data sets are shrunk towards zero when combined. We
continue with the imputed data from the MICE Default below.
Table IV presents the results from the stepwise model selection procedure after multiple imputation. When
performing AIC-based model selection on each imputed dataset, the procedure never selects gender (gndr)
which is why its coefficient is shrunk completely to zero. This contrasts with a small non-zero coefficient in the
standard imputation-analysis and an even larger coefficient in the complete case analysis (Table III). Education
(eduyrs) is also of little importance when predicting our outcome variable and is shrunk very close to zero.
Notably, the coefficient of the interaction term is no longer significant at an alpha of 5% as the 95% confidence
interval includes zero.
Table IV: Combining stepwise selection and multiple imputation
Variable Estimate Std.Error Lower CI Upper CI
Intercept 3.62484 0.38672 2.86673 4.38296
flapppl 0.29250 0.01182 0.26932 0.31567
agea 0.01202 0.00419 0.00381 0.02023
gndr 0.00000 0.00000 0.00000 0.00000
health2 -0.34627 0.09233 -0.52723 -0.16530
health3 -0.81544 0.09090 -0.99360 -0.63728
health4 -1.56644 0.10127 -1.76493 -1.36795
health5 -2.26402 0.13737 -2.53335 -1.99469
eduyrs -0.00042 0.00229 -0.00505 0.00422
hinctnta 0.08577 0.01064 0.06489 0.10664
grdfincc 0.07598 0.00858 0.05916 0.09281
income_int -0.00742 0.00392 -0.01516 0.00032
immi 0.03221 0.00359 0.02516 0.03925
Notes: Country fixed effects are included. Calculated confidence intervals are 95% intervals.
Model selection and averaging on imputed data is an area of on-going research. It is not advised to perform
model selection on the complete cases, but rather use the desired selection method based on Rubin’s (1987)
rules (Chen & Wang 2013). In our example, performing stepwise selection on the complete data does indeed
exclude gender, but estimates education to have a coefficient around 23 times larger ( = -0.01, p = 0.14) than
on the imputed data ( = -0.0004, p = 0.85). As especially model averaging may be impractical for complicated
models, e.g. with numerous possible interactions, stacking the imputed data sets into a single data set using
weights at the model-building state may be a pragmatic alternative (Wood et al. 2008). Simply averaging
model selection criteria such as AIC across imputed data set is temptingly simple, but not supported in the MI
literature (Schomaker & Heumann 2014; White et al. 2011).
3.5.3 Reporting results
Reporting the regression results based on imputed data is not different from regression results based on
regular data. However, when using multiple imputation to account for missing data, the data analyst must
supply additional information either in the main paper / report or in the supplement materials. Because many
of the decisions we make when imputing data may affect the end-result, it is reasonable to state these
decisions. In praxis, there seems to be deficiencies in the documentation of missing data and the details about
the imputation (Hayati Rezvan et al. 2015). Sterne et al. (Sterne et al. 2009) provide a set of guidelines about
which information to report. Below we condense those that should be made as a minimum and refer to the
paper for further details.
(1) The software used including the key settings described in this paper as e.g. number of imputed datasets. (2)
A list of the variables used in the imputation model incl. auxiliary variables (e.g. Table I). (3) Handling of non-
normally distributed variables and non-linear terms. (4) Comparison of observed and imputed values for
variables with a high level of missing rates (e.g. Figure 3). (5) Discussion of notable differences between
analyses of complete cases and imputed data. Finally, (6) a summary of the number and fraction of missing
values and additional relevant information of the way the data are missing.
4. Concluding remarks
MI is among the most prominent methods to handle missing data and its superiority has been documented in
a long range of studies. One of the hurdles of getting started with MI is that it may be difficult navigating
around the pitfalls. With this overview, we aim to help more researchers to get started with implementing MI-
techniques such as MICE instead of inferior approaches. We hope the data analysts who already actively apply
MICE may use it to review some of the newest developments.
We chose a practical focus on fewer techniques in one selected software environment to keep the guidelines
simple. We focused on MICE but fully recognize maximum likelihood routines as e.g. full maximum likelihood.
MICE and full maximum likelihood produce identical results making the choice between them a matter of
personal preference (Enders et al. 2014; Schafer 2003; Collins et al. 2001). We also focused on the R statistical
environment in the provided example. For an overview of alternative software packages, we recommend
consulting Horton and Kleinman (2007). Finally, we also focused on a subset of commonly used models in the
analysis example. After having been introduced to our suggested guidelines, we hope that readers are
motivated to dig into some of the details with regard to more complicated analyses of e.g. hierarchical (Zhao &
Yucel 2009; van Buuren 2011; Enders et al. 2016; Grund et al. 2016) or longitudinal data structures (Ibrahim &
Molenberghs 2009; Jansen et al. 2006).
We hope that our synthesis of the literature into guidelines may help improve current standards of handling
missing data. We end with the caveat that our practical guidelines are no more than the name suggests. We
have tried to condense the statistical correct praxis into discrete rules-of-thumb to make it accessible to a
larger audience. While nuances and details are often lost in a summary, we believe that accessible guidelines
are preferable to the praxis that is currently dominating. More detailed and complex decision-rules for praxis
are certainly possible but in our opinion, their contribution will be marginal if the basics are ignored.
References
Azur, M.J. et al., 2011. Multiple imputation by chained equations: what is it and how does it work? International journal of
methods in psychiatric research, 20(1), pp.40–9.
Bartlett, J.W. et al., 2014. Multiple imputation of covariates by fully conditional specification: Accommodating the
substantive model. Statistical methods in medical research, 24(4), pp.462–487.
Bodner, T.E., 2008. What Improves with Increased Missing Data Imputations? Structural Equation Modeling: A
Multidisciplinary Journal, 15(4), pp.651–675.
Burgette, L.F. & Reiter, J.P., 2010. Multiple imputation for missing data via sequential regression trees. American journal of
epidemiology, 172(9), pp.1070–6.
van Buuren, S., 2011. Multiple imputation of multilevel data. In J. Hox & J. K. Roberts, eds. The Handbook of Advanced
Multilevel Analysis. Milton Park: Routledge, pp. 173–196.
van Buuren, S. & Groothuis-Oudshoorn, K., 2011. mice: Multivariate Imputation by Chained Equations in R. Journal of
Statistical Software, 45(3), pp.1–67.
Chen, Q. & Wang, S., 2013. Variable selection for multiply-imputed data with application to dioxin exposure study.
Statistics in medicine, 32(21), pp.3646–59.
Chevret, S., Seaman, S. & Resche-Rigon, M., 2015. Multiple imputation: a mature approach to dealing with missing data.
Intensive care medicine, 41(2), pp.348–50.
Collins, L.M., Schafer, J.L. & Kam, C.M., 2001. A comparison of inclusive and restrictive strategies in modern missing data
procedures. Psychological methods, 6(4), pp.330–51.
Donders, A.R.T. et al., 2006. Review: a gentle introduction to imputation of missing values. Journal of clinical epidemiology,
59(10), pp.1087–91.

Doove, L.L., Van Buuren, S. & Dusseldorp, E., 2014. Recursive partitioning for missing data imputation in the presence of
interaction effects. Computational Statistics & Data Analysis, 72, pp.92–104.
Enders, C.K., Baraldi, A.N. & Cham, H., 2014. Estimating interaction effects with incomplete predictor variables.
Psychological methods, 19(1), pp.39–55.
Enders, C.K., Mistler, S.A. & Keller, B.T., 2016. Multilevel multiple imputation: A review and evaluation of joint modeling
and chained equations imputation. Psychological Methods, 21(2), pp.222–240.
ESS Round 6: European Social Survey Round 6 Data, 2012. Data file edition 2.1,
Fichman, M. & Cummings, J.N., 2003. Multiple imputation for missing data: Making the most of what you know.
Organizational Research Methods, 6(3), pp.282–308.
Gottschall, A.C., West, S.G. & Enders, C.K., 2011. A Comparison of Item-Level and Scale-Level Multiple Imputation for
Questionnaire Batteries. Multivariate Behavioral Research, 47(1), pp.1–25.
Graham, J.W., 2009. Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, pp.549–576.
Graham, J.W., Olchowski, A.E. & Gilreath, T.D., 2007. How Many Imputations are Really Needed? Some Practical
Clarifications of Multiple Imputation Theory. Prevention Science, 8(3), pp.206–213.
Graham, J.W. & Schafer, J.L., 1999. On the performance of multiple imputation for multivariate data with small sample size.
In R. Hoyle, ed. Statistical strategies for small sample research. Thousand Oaks, CA: Sage, pp. 1–29.
Greenland, S. & Finkle, W.D., 1995. A critical look at methods for handling missing covariates in epidemiologic regression
analyses. American journal of epidemiology, 142(12), pp.1255–64.
Grund, S., Lüdtke, O. & Robitzsch, A., 2016. Multiple imputation of missing covariate values in multilevel models with
random slopes: a cautionary note. Behavior Research Methods, 48(2), pp.640–649.
Haji-Maghsoudi, S. et al., 2013. Influence of pattern of missing data on performance of imputation methods: an example
using national data on drug injection in prisons. International journal of health policy and management, 1(1), pp.69–
77.
Hardt, J., Herke, M. & Leonhart, R., 2012. Auxiliary variables in multiple imputation in regression with missing X: a warning
against including too many in small sample research. BMC medical research methodology, 12(1), p.184.
Harel, O., 2009. The estimation of R2 and adjusted R2 in incomplete data sets using multiple imputation. Journal of Applied
Statistics, 36(10), pp.1109–1118.
Hastie, T., Tibshirani, R. & Friedman, J., 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction
2nd ed., Springer.
Hayati Rezvan, P., Lee, K.J. & Simpson, J.A., 2015. The rise of multiple imputation: a review of the reporting and
implementation of the method in medical research. BMC medical research methodology, 15(1), p.30.
He, Y. et al., 2010. Multiple imputation in a large-scale complex survey: a practical guide. Statistical methods in medical
research, 19(6), pp.653–70.
Heitjan, D.F. & Rubin, D.B., 1990. Inference from Coarse Data via Multiple Imputation with Application to Age Heaping.
Journal of the American Statistical Association, 85(410), pp.304–314.
von Hippel, P.T., 2009. How to impute interactions, squares, and other transformed variables. Sociological Methodology,
39(1), pp.265–291.
Horton, N.J. & Kleinman, K.P., 2007. Much ado about nothing: A comparison of missing data methods and software to fit
incomplete data regression models. The American statistician, 61(1), pp.79–90.
Ibrahim, J.G. & Molenberghs, G., 2009. Missing data methods in longitudinal studies: a review. TEST, 18(1), pp.1–43.
Jansen, I. et al., 2006. Analyzing Incomplete Discrete Longitudinal Clinical Trial Data. Statistical Science, 21(1), pp.52–69.
Janssen, K.J.M. et al., 2010. Missing covariate data in medical research: to impute is better than to ignore. Journal of clinical
epidemiology, 63(7), pp.721–7.
Kenward, M.G. & Carpenter, J., 2007. Multiple imputation: current perspectives. Statistical methods in medical research,
16(3), pp.199–218.
Kim, J.K. et al., 2006. On the bias of the multiple-imputation variance estimator in survey sampling. Journal of the Royal
Statistical Society: Series B (Statistical Methodology), 68(3), pp.509–521.
King, G. et al., 2001. Analyzing incomplete political science data: an alternative algorithm for multiple imputation. American
Political Science Review, 95(1), pp.49–69.
Knol, M.J. et al., 2010. Unpredictable bias when using the missing indicator method or complete case analysis for missing
confounder values: an empirical example. Journal of Clinical Epidemiology, 63(7), pp.728–736.
Lee, K.J. & Carlin, J.B., 2012. Recovery of information from multiple imputation: a simulation study. Emerging themes in
epidemiology, 9(1), p.3.
Little, R.J.A., 1988. Missing-data adjustments in large surveys. Journal of Business & Economic Statistics, 6(3), pp.287–296.
Little, R.J.A., 1995. Modeling the Drop-Out Mechanism in Repeated-Measures Studies. Journal of the American Statistical
Association, 90(431), pp.1112–1121.
Little, R.J.A. & Rubin, D.B., 1987. Statistical analysis with missing data, New York: John Wiley.
Marshall, A., Altman, D.G. & Holder, R.L., 2010. Comparison of imputation methods for handling missing covariate data
when fitting a Cox proportional hazards model: a resampling study. BMC medical research methodology, 10(1), p.112.
Meng, X.-L. & Rubin, D.B., 1992. Performing likelihood ratio tests with multiply-imputed data sets. Biometrika, 79(1),
pp.103–111.
Morris, T.P., White, I.R. & Royston, P., 2014. Tuning multiple imputation by predictive mean matching and local residual
draws. BMC medical research methodology, 14(1), p.75.
Newman, D.A., 2014. Missing data: Five practical guidelines. Organizational Research Methods, 17(4), pp.372–411.
Posada, D. & Buckley, T.R., 2004. Model selection and model averaging in phylogenetics: advantages of akaike information
criterion and bayesian approaches over likelihood ratio tests. Systematic biology, 53(5), pp.793–808.
R Core Team, 2015. R: A Language and Environment for Statistical Computing.
Roth, P., 1994. Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47(3), pp.537–560.
Rubin, D.B., 1987. Multiple imputation for nonresponse in surveys, New York: John Wiley and Sons.
Rubin, D.B., 1996. Multiple imputation after 18+ years. American Statistical Association, 91(434), pp.473–489.
Rubin, D.B. & Schenker, N., 1986. Multiple Imputation for Interval Estimation from Simple Random Samples with Ignorable
Nonresponse. Journal of the American Statistical Association, 81(394), pp.366–374.
Schafer, J.L., 1997. Analysis of incomplete data, London: Chapman & Hall.
Schafer, J.L., 1999. Multiple imputation: a primer. Statistical Methods in Medical Research, 8(3), pp.3–15.
Schafer, J.L., 2003. Multiple imputation in multivariate problems when the imputation and analysis models differ. Statistica
Neerlandica, 57(1), pp.19–35.
Schenker, N. et al., 2006. Multiple Imputation of Missing Income Data in the National Health Interview Survey. Journal of
the American Statistical Association, 101(475), pp.924–933.
Schenker, N. & Taylor, J.M.G., 1996. Partially parametric techniques for multiple imputation. Computational Statistics &
Data Analysis, 22(4), pp.425–446.
Schomaker, M., 2015. MAMI: Model averaging and model selection after multiple imputation.
Schomaker, M. & Heumann, C., 2014. Model selection and model averaging after multiple imputation. Computational
Statistics & Data Analysis, 71, pp.758–770.
Seaman, S.R., Bartlett, J.W. & White, I.R., 2012. Multiple imputation of missing covariates with non-linear effects and
interactions: an evaluation of statistical methods. BMC medical research methodology, 12(1), p.46.
Shah, A.D. et al., 2014. Comparison of random forest and parametric imputation models for imputing missing data using
MICE: a CALIBER study. American journal of epidemiology, 179(6), pp.764–74.
Stekhoven, D.J. & Bühlmann, P., 2012. MissForest--non-parametric missing value imputation for mixed-type data.
Bioinformatics (Oxford, England), 28(1), pp.112–8.
Sterne, J.A.C. et al., 2009. Multiple imputation for missing data in epidemiological and clinical research: potential and
pitfalls. BMJ (Clinical research ed.), 338, p.b2393.
Stuart, E.A. et al., 2009. Multiple imputation with large data sets: a case study of the Children’s Mental Health Initiative.
American journal of epidemiology, 169(9), pp.1133–9.
White, I.R. & Royston, P., 2009. Imputing missing covariate values for the Cox model. Statistics in medicine, 28(15),
pp.1982–98.
White, I.R., Royston, P. & Wood, A.M., 2011. Multiple imputation using chained equations: Issues and guidance for
practice. Statistics in Medicine, 30(4), pp.377–399.
Wood, A.M., White, I.R. & Royston, P., 2008. How should variable selection be performed with multiply imputed data?
Statistics in medicine, 27(17), pp.3227–46.
Zhao, E. & Yucel, R.M., 2009. Performance of sequential imputation method in multilevel applications. In American
Statistical Association. Alexandria, pp. 2800–2810.
Zhao, Y. & Long, Q., 2016. Multiple imputation in the presence of high-dimensional data. Statistical Methods in Medical
Research, 25(5), pp.2021–2035.


S3.1 EJBRM-volume-15-issue-1 (4) QL (3) QT

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

S3.1 EJBRM-volume-15-issue-1 (4) QL (3) QT

Uploaded by

Copyright:

Available Formats

Editorial for EJBRM Volume 15 Issue 1

This issue has four research papers –

ISSN 1477-7029 1 ©ACPIL

Jeremy Rose and Christian Lennerholt

2. A design science approach to process building

Design science step Our research approach

3. Problem awareness: costs of Qualitative research, advances in text mining

 empirical material that is not easily quantifiable

www.ejbrm.com 3 ISSN 1477-7029

 making contact and interacting with research subjects and organisations

 resource finding (retrieving relevant documents)

 elicitation (finding and downloading relevant data)

 Problematisation - formulating a research problem (objective or question) that can be solved

5. Development: five process steps in the text mining literature

www.ejbrm.com 5 ISSN 1477-7029

In respect to representativeness, Krippendorff (2004) identifies a variety of sampling techniques as appropriate

www.ejbrm.com 7 ISSN 1477-7029

 which analysis task is appropriate for the research problem?

www.ejbrm.com 9 ISSN 1477-7029

6.4 Analysing text

 performance tags business analytics

www.ejbrm.com 11 ISSN 1477-7029

7. Results: a generic process for low cost text mining

www.ejbrm.com 13 ISSN 1477-7029

www.ejbrm.com 15 ISSN 1477-7029

Appendix 1. sample blog download:

David A.L Coldwell

ISSN 1477-7029 17 ©ACPIL

2. Background to the problem: information deluge, specialization and methodological

www.ejbrm.com 19 ISSN 1477-7029

3. Bridging the methodological divide: the place of multidisciplinary multinational

• Lack of methodological concensus among practicising scientists

4. Benefits of multi-disciplinary and multicultural collaborative research: A case study of

www.ejbrm.com 21 ISSN 1477-7029

Network arose Project arose Group Division Project Collaboration Project

www.ejbrm.com 23 ISSN 1477-7029

www.ejbrm.com 25 ISSN 1477-7029

6. Summary and Conclusion

www.ejbrm.com 27 ISSN 1477-7029

Table 1: The Profile of the Informants

3.2 The research behind the theoretical conjecture

4. What is a Knowledge Café and how can it contribute to research?

www.ejbrm.com 31 ISSN 1477-7029

Figure 2: Application of the Knowledge Café

5. The use of the Knowledge Café in the research project

e-Government applications are driven by a need to modernise government processes in the

www.ejbrm.com 33 ISSN 1477-7029

www.ejbrm.com 35 ISSN 1477-7029

• Basic elements of the approach,

www.ejbrm.com 37 ISSN 1477-7029

7. Conclusion and summary

The success of a Knowledge Café as described here is contingent on:

• Being able to attract a number of appropriately knowledgeable informants to participate in the

Appendix A: Summary of the transcript of the plenary group meeting

www.ejbrm.com 39 ISSN 1477-7029

Jesper N. Wulff1 and Linda Ejlskov2

ISSN 1477-7029 41 ©ACPIL

Figure 1: The MICE-process (Rubin 1987)

www.ejbrm.com 43 ISSN 1477-7029

Figure 2: MICE decision tree

3. Multiple imputation: An example

Analysis model variables

www.ejbrm.com 45 ISSN 1477-7029