Information Systems

Department
College of Computing
Sciences
New Jersey Institute
of Technology
Research Paper Review - Structure and
ormat !uidelines
Michael Bieber
This set of guidelines was originally designed for a course, so it
includes grading guidelines, which most readers can ignore.
Notes:
1. Follow the guidelines here on structuring and formatting your review.
2. Include each category in a separate section or subsection. Ensure that you discuss
each category! If a category does not apply! then actually write in that
sub"section that the category does not apply." #e will be loo$ing for each
category when grading.
%. &eference other articles you've read! if appropriate. (ust as in the articles you've
read! include a reference mar$er where you refer to each in your review! and
provide full bibliographic references in a bibliography section at the end of your
report.
). *our final report should be ) full pages or more if measured in +imes ,ew
&oman 1-pt single spaced"! which is e.uivalent to /01- pages in most #eb
browsers. +his does not include your bibliography! citation analysis! diagrams or
large gaps between parts of your report. +here is no ma1imum length.
2. 3o not copy any part of the article into your review. If you want to use more than
%0) of the author's words! then use .uotation mar$s! and add a page number from
the article if available" when you cite this .uotation.
4. 5roof0read your review for grammar and clarity as well as spelling. 6pell chec$ers
do not fi1 grammar. *ou should be sure that you have caught all mista$es and
written clearly. Most word processors have a grammar chec$er 0 use it!
/. For the printed copy of your review! turn in 7ust a plain stapled blac$ 8 white
copy of your review. 3o not put your review in a folder. 9void fancy covers!
colors or fonts. #e want to 7udge your content! not visual presentation.
:. 5lease reread the note on plagiarism posted in the syllabus and on the course #eb
site.

"dditional References to #utside $aterials
In sections 6-14, utilize at least 3 citations to articles outside those on
the class reading list. se these articles to !ac" u# your analysis, to
descri!e or address issues, im#lications or recommendations for
further research or de$elo#ment.

!rading Notes
%e ha$e included the grading criteria in these guidelines.
nless otherwise noted, the categories are:
great; fully discussed! e1plained well and well07ustified
o$; partial discussion< not e1plained well or well07ustified
poor; barely discussed
=ero; not discussed
%ength !rading
%e ta"e & #oints o' for each full #age missing (i.e., less than 4 full
#ages) and 3 #oints o' for e$ery half #age missing. Note that large
ga#s !etween sections will not count towards the full #age length.
&diting !rading
*as the re$iew !een #rooofread for !oth s#elling and grammar+ If not
u# to 3 #oints o' for #oor grammar and u# to 3 #oints o' for #oor
s#elling.

Preparing for 'our Review
The #ur#ose of research articles is to #resent new or re,ne conce#tual
ideas, or to #resent new e$idence for conce#tual ideas. - research
article re$iew results from critically e.amining a research article. /ou
will ha$e to read your article several times to understand it fully
enough to re$iew #ro#erly.
0ften, com#aring your article to others will hel# you determine its
1uality.
-lso, thin" a!out the article and its research ideas in terms of each of
the di'erent wee"2s conce#ts and framewor"s we study in class. 3o
the ideas in the article ,t all the categories of a##ro#riate class
conce#ts, models and framewor"s+ This often will hel# you see things
the authors missed, thin" a!out things the authors write in a new light,
or see that the authors indeed co$ered a to#ic thoroughly.

#utline of the "rticle Review
4lease include the following categories in your article re$iew.
1. Full Bibliographic &eference (-3 if missing)
2. Introduction; >b7ectives! 9rticle 3omain! 9udience! (ournal and
?onceptual@Emprical ?lassification (8)
%. Aery Brief 6ummary (4)
). &esults (8)
2. ?lass &eadings (4)
4. ?ontributions (8)
/. Foundation (4)
:. 6ynthesis with ?lass Materials (12 +8 extra credit)
B. 9nalysis 8 9dditional 9nalysis (4 +8 extra credit)
1-. Ceneral ?riti.ue (10 + 6 extra credit)
11. Further ?riti.ue of a ?onceptual 9rticle 0or0 (12)
Further ?riti.ue of an Empirical 9rticle (12+2 extra credit)
12. Issues listed by the author" (6+3 extra credit)
1%. Issues in your opinion" (6+6 extra credit)
1). Impact (9)
12. Duestions (6)
14. 9nnotated Bibliography (-5 if missing)
1/. ?itation 9nalysis 9ppendi1 (6)
In addition, you can lose #oints if your re$iew is too short or #oorly
edited. 5ee the 6rading Notes section a!o$e.
() ull *i+liographic Reference
5tate the full !i!liogra#hic reference for the article you are re$iewing
(authors, title, 7ournal name, $olume, issue, year, #age num!ers, etc.)
Im#ortant: this is not the !i!liogra#hy listed at the end of the article,
rather the citation of the article itself8
Grading: -3 if missing
,) Introduction- #+.ectives/ "rticle
Domain/ "udience/ Journal and
Conceptual0&mprical Classi1cation
Note: For the on-line reviews done in some class sections, this
category may be broken up into several separate subcategories. For
the written review, please discuss all of these subcategories together
as follows.
4aragra#h 1: 5tate the o!7ecti$es (goals or #ur#ose) of the article.
%hat is the article2s domain (to#ic area)+
4aragra#h 9:
9udience; 6tate the article's intended audience. 9t what level is it written! and
what general bac$ground should the reader have< what general bac$ground
materials should the reader be familiar with to understand the articleE
9ppropriate (ournalE; #hy is the 7ournal appropriate or inappropriate" for this
articleE ?hec$ the mission statement or purpose of the 7ournal itself from its
cover or its #eb site."
4aragra#h 3: 5tate whether the article is :conce#tual: or :em#irical:,
and why you !elie$e it is conce#tual or em#irical. ;m#irical articles
and conce#tual articles ha$e a similar o!7ecti$e: to su!stantiate an
argument #ro#osed !y the author. %hile a conce#tual article su##orts
such an argument !ased on logical and #ersuasi$e reasoning, an
em#irical article o'ers em#irical e$idence to su##ort the argument.
;m#irical articles o'er su!stantial, detailed e$idence which the authors
analyze using statistical methods. ;m#irical articles must include
hy#otheses (or #ro#ositions), detailed research results, and (statistical)
analyses of this em#irical e$idence. ;m#irical research includes
e.#eriments, sur$eys, 1uestionnaires, ,eld studies, etc, and to limited
degree, case studies. <once#tual articles may refer to such em#irical
e$idence, !ut do not #ro$ide the detailed analysis of that e$idence. 0f
course, !oth ty#es of articles can use real life e.am#les to !ac" u#
their #oints. =ust !ecause an article #ro$ides e.am#les, does not
necessarily mean that it is em#irical. (The lesson to ta"e home is not to
consider a conce#tual article to !e an em#irical one 7ust !ecause it
#ro$ides some summarized or some unanalyzed data.)
Grading: b!ectives: great - 3" ok - #" poor - $
Grading: %udience&'ournal %ppropriateness: great - 3" ok - #" poor - $
Grading: (onceptual vs. empirical: great - #" ok&poor - $
2) 3ery *rief Summary
>or our article re$iews, we do not want you to s#end much s#ace
summarizing the article. Instead we are more interested in your
analysis of the article.
Thus, in this section, summarize the article only $ery !rie?y (9-3
#aragra#hs).
If #ossi!le, use the I5 research #aradigm as the format of your
summary, !ut remaining $ery !rief:
5aragraph 1; what is the problem or opportunity being addressed
5aragraph 2; which solution is proposed the solution could be a new model or a
theory that e1plains the problem"
5aragraph %; what evidence is put forth that this solution is appropriate If this is
an empirical article! be sure to briefly describe what $ind of empirical study was
done as part of the evidence"
Grading: great - )" ok - #" poor - $
4) Results
@ery !rie?y summarize the im#ortant #oints (o!ser$ations,
conclusions, ,ndings) and :ta"e home messages: in the article.
4lease do not re#eat lists of items in the articles - 7ust summarize the
essence of these if you feel they are necessary to include.
Grading: great - *" ok - +" poor - #
5) Class Readings
1. 3oes this article directly cite any of the class readings, i.e., does any
class reading a##ear e.#licitly in its !i!liogra#hy or reference section+
If not, state this e.#licitly. If so, clearly descri!e how the authors use
the cited article. *ow does the article you are re$iewing relate to
andAor !uild u#on the class article it cites+
If this article does not cite any class readings then 7ust state this. ,-f
you do not state this e.plicitly, you will not receive credit for this
section./ 3o not discuss any other readings, such as other readings on
the same to#ic or !y the same author. 5a$e any discussions of similar
articles for your synthesis section !elow.
9. 3o any of the class readings cite your article (!esides the te.t!oo")+
If so, clearly descri!e how.
If no class readings cite your article, then write in your re$iew :No class
readings cite this article.: ,-f you do not state this e.plicitly, you will
not receive credit for this section./
Be sure to add all references you cite to the !i!liogra#hy.
Grading: great - )" ok - #" poor - $ 0-f none, then score ) by default if
this has been stated e.plicitly.1
6) Contri+utions
-n article ma"es a :contri!ution: !y adding to the "nowledge of
researchers in a research ,eld. -n article can ma"e a contri!ution to
the research ,eld in many ways. 3oes it #ro$ide a new way to loo" at a
#ro!lem+ 3oes it !ring together or :synthesize: se$eral conce#ts (or
framewor"s, models, etc.) together in an insightful way that has not
!een done !efore+ 3oes it #ro$ide new solutions+ 3oes it #ro$ide new
results+ 3oes it identify new issues+ 3oes it #ro$ide a com#rehensi$e
sur$ey or re$iew of a domain+ 3oes it #ro$ide new insights+
-lso, is it salient (rele$ant and current) to a #articular scienti,c issue or
managerial #ro!lem+ -re the issues addressed introduced in a way
that their rele$ance to #ractice is e$ident+ %ould answers to the
1uestions raised in the article li"ely to !e useful to researchers and
managers+
Note: 3o not discuss the contri!utions of the technologies the article
descri!es, !ut rather the contri!utions of the article itself8
The article2s contri!utions should !e original. To the !est of your
"nowledge, are they+ -re the article2s ta"e-home messages new+
3escri!e each contri!ution clearly in a se#arate #aragra#h or !ullet
#oint. 3iscuss why the contri!ution is im#ortant.
-lternati$ely, if you !elie$e the article ma"es no contri!utions, e.#lain
why clearly.
Grading: great - *" ok - +" poor - #
7) oundation
6ood research often is !uilt u#on theories and framewor"s that other
researchers ha$e de$elo#ed. 5ometimes articles will !e su!stantially
!ased u#on this #rior wor", and refer !ac" to it in some detail. (Not all
research articles will do this.)
%hich theoretical foundations does this article and research !uild on, if
any+ In what ways+ Include referencesAcitations of the foundation wor".
(/ou can determine this in #art from the wor"s the article cites.)
Note, howe$er, that most wor"s cited are not core foundational wor",
!ut rather 7ust su##ort certain as#ects of the article. 5imilarly, do not
confuse a general discussion of related to#ics as foundational wor".
If the article does not !uild u#on "ey #ieces of #rior research, then
write in your re$iew :This article does not !uild u#on any foundation
research.: ,-f you do not state this e.plicitly, you will not receive credit
for this section./
Grading: great - )" ok - 3" poor -$ 0-f none, then score ) by default if
this has been stated e.plicitly1
8) Synthesis with Class $aterials
The synthesis section should !e at least one full #age.
5ynthesis means analyzing a #articular to#ic !y com#aring and
contrasting it with, and thin"ing a!out it from the $iew#oint of, the
class materials from across the semester. These materials include the
articles, models, framewor"s, guidelines and other conce#ts we2$e
co$ered. (0f course, only certain materials will !e rele$ant for any
gi$en article.)
Note: /ou ha$e to do this synthesis8 /ou need to relate this article to
other things we ha$e studied, so !y de,nition you will not ,nd this
analysis in the article itself8
3iscuss the article2s research ideas and results in terms of any rele$ant
materials co$ered in class or which you ha$e found in the readings. /ou
can also chec" the conce#ts in the :to "now: lin" on the :1uic" lin"s:
#ortion of the course %e! site. <ite these readings e.#licitly, including
their source in the !i!liogra#hy and a !i!liogra#hic mar"er in the te.t
(e.g., CTuro' et al., 1DDDE).
/ou also could analyze the a##roach the author too" to the article2s
analysis and discussion. 3iscuss the article2s a##roach and results in
terms of one or more of the framewor"s, etc., from the te.t or
readings, or any you ,nd elsewhere. >or e.am#le, if the authors
discuss any ty#e of information system, you could use -lter2s %<-
analysis to e.amine how they a##roached that information system. Try
to do this for all the models and framewor"s, etc., which a##ly to your
article.
-s #art of this analysis, reference other articles you2$e read, when
a##ro#riate. <om#are the a##roach, results and contri!ution with all
articles a!out similar to#ics or with a similar a##roach. >or e.am#le, if
your article de$elo#s a new framewor", com#are it with
Bandyo#adyhah2s framewor" criteria (and $ice $ersa - whoe$er does
Bandyo#adyhah2s article could test his criteria on framewor"s from the
other readings). Include any articles you cite in the !i!liogra#hy and
use !i!liogra#hic mar"ers in the te.t.
>or all of these, do your synthesis com#arison in as much de#th as you
can8
Grading: four items up to #2 points total ,$# points plus * points e.tra
credit/ - for each item: great - + ok - #" poor - $
Great: discussed deeply and relating the article in detail with the
synthesi3ed models and frameworks.
4: the synthesi3ed information is only discussed in general
9) "nalysis
Note: 5any people assume this category is the same as 6General
(riti7ue6. -t is not. General (riti7ue is a di8erent category from this,
and follows below.
%hat has changed since the article was written+ *ow do it2s lessons,
ideas and theories still a##ly+ To what e.tent has its issues !een
resol$ed+
Grading: great - )" ok - #" poor - $
"dditional "nalysis
0#tionally, try a##lying the article2s models, framewor"s and
guidelines, etc. yourself. 3o you ,nd them useful+
In addition, you may o#tionally add your own additional analysis in a
se#arate su!section. (3o not re#eat the author2s analysis in the #a#er -
you could summarize this as #art of the results section.)
Grading: this section is e.tra credit only: great - *" ok - +" poor - #
(:) !eneral Criti;ue
In this section you should state your o#inions of how well (or #oorly)
the authors did their research and #resented the research results in the
article. /our criti1ue can contain !oth #ositi$e and negati$e comments.
=ustify and e.#lain in detail each of your criti1ue #oints in a se#arate
#aragra#h of at least 4-& sentences.
The following are suggestions only:
3oes it build upon the appropriate foundation i.e.! upon appropriate prior
research"E
3id the authors choose the correct approach! and then e1ecute it properlyE
Fow confident are you in the article's results! and whyE
9re its ideas really new! or do the authors simply repac$age old ideas and perhaps
give them a new nameE
3o the authors discuss everything they promise in the article's introduction and
outlineE
#hat are the article's shortcomings faults" and limitations boundaries"E 3id it
discuss all of the important aspects and issues in its domain topic area"E
In what way should the article have made a contribution! but then did notE
3o the authors ma$e appropriate comparisons to similar events! cases or
occurrencesE
Fow complete and thorough a 7ob did the authors doE 3o the authors include an
ade.uate discussion! analysis and conclusionsE 3id they 7ustify everything
ade.uatelyE 3id they provide enough bac$ground information for the intended
audience to understand itE For you to understand itE
#ere there ade.uate and appropriate e1amples and illustrationsE
>or full credit, as" yourself these 1uestions when 7ustifying your
criti1ue #oints:
why@why notE
howE
what distinguishes the differences@different approaches! and in what waysE
Grading: four items up to $9 points total ,$2 points plus 9 points e.tra
credit/ - for each item: great - )" ok - #" poor - $
((c) urther Criti;ue of a Conceptual
"rticle
FFF only for conce#tual articles
0adapted from guidelines from :r. :an ;obey, Georgia <tate
=niversity1
- criti1ue of a conce#tual article e.amines the logic of the arguments
made !y the authors. Both strengths and wea"nesses should !e
identi,ed in a criti1ue. ;.#lain and 7ustify each of your criti1ue #oints
in at least 3-4 sentences. 6i$e e.am#les whene$er #ossi!le.
To the !est of your a!ilities, discuss each of the following categories in
a se#arate #aragra#h:
1. G06I<-G <0N5I5T;N</: 3o any #arts of the article or research
contradict or in$alidate other #arts+ If so, ha$e the authors
ac"nowledged and e.#lained this ade1uately+
9. <0*;H;N<;: 3oes the article ma"e sense+ 3id the authors
a##roach this article (and this research) sensi!ly+ 3oes the article
de$elo# an argument that follows a coherent line of reasoning+ -re the
!oundaries of the argument reasona!ly well de,ned+ 3oes the
argument antici#ate most, if not all, ri$al arguments+
3oes the article ?ow in a logical se1uence+ 3o later #arts !uild
logically u#on earlier #arts+
3. 5B5T-N<;: 3oes the article #ro$ide an argument or a line of
reasoning that o'ers insight into im#ortant issues, or does it merely
summarize #re$ious studies in a shallow way that does not re?ect
de#th of analysis+ 3oes the article #ro$ide ways (a model, framewor",
guidelines, etc.) to guide future thin"ing a!out the issue(s) the author
is addressing+
4. >0<5: Is there a clear audience that the authors address+ %as the
article written at the a##ro#riate le$el for this audience+
Grading: for each: great - 3 ok - #" poor - $
((e) urther Criti;ue of an &mpirical
"rticle
FFF only for em#irical articles
0adapted from guidelines from :r. :an ;obey, Georgia <tate
=niversity1
- criti1ue of an em#irical article e.amines the strength of the em#irical
e$idence su##orting the author2s argument. Both strengths and
wea"nesses should !e identi,ed in a criti1ue. ;.#lain and 7ustify each
of your criti1ue #oints in at least 3-4 sentences.
To the !est of your a!ilities, discuss each of the following categories in
a se#arate #aragra#h:
1. <G-HIT/: Is the article2s #ur#ose and argument clear+ 3o the
researchers clearly de$elo# a ma7or research 1uestion, #ro#osition, or
hy#othesis that is to !e e$aluated in the em#irical study and discussed
in this article+ If the study is e.#loratory (#reliminary), is suIcient
7usti,cation for an e.#loratory strategy gi$en+
9. T*;0H;TI<-G 6H0N3IN6: Is the researcher2s argument grounded
in more !asic theory+ Is it clear whether the structure of the em#irical
study (i.e., what they do) was deri$ed from theory, or 7ust made u#+ In
theory-!uilding articles, is the need for new theory ade1uately
esta!lished+
3. 3;5I6N 0> H;5;-H<* IN@;5TI6-TI0N: Is it clear e.actly how the
em#irical study was carried out+ Is the design of the research a##roach
(,eld study, e.#eriments, 1uestionnaires, etc. - !oth contents and how
they will !e used) ade1uate to address the common threats to internal
and e.ternal $alidity+ *a$e a##ro#riate controls !een esta!lished, and
is the selection of research sites 7usti,ed+ -re the hy#otheses and
e.#eriments, etc., signi,cant+
4. J;-5H;J;NT: ;m#irical studies can ha$e 1uantitati$e
measurements (i.e., numeric results) and 1ualitati$e or su!7ecti$e
measurements. -re the measures used ade1uately descri!ed (i.e.,
what is measured in the study and how)+ -re data on the relia!ility and
$alidity of these measures re#orted+ 3oes the article feel anecdotal or
solidly su##orted with e$idence+ >or e.am#le, in case or ,eld studies,
are the results well documented+ Is it clear who the su!7ects were, and
with whom inter$iews were carried out+ %ere im#ortant results cross-
chec"ed, i.e., determined across a range of su!7ects or 7ust gotten from
one or two su!7ects+
&. -N-G/5I5: Is the analysis of em#irical data conducted #ro#erly+ 3o
the data conform to the re1uirements of any statistical tests used+ -re
1ualitati$e data ade1uately descri!ed and #resented+
6. 3I5<55I0N -N3 <0N<G5I0N5: In discussing the results of the
em#irical study, do the authors remain true to the actual ,ndings of
the study+ -re the claims made in the conclusion of the article actually
su##orted !y the em#irical data+ If the study is e.#loratory, do the
authors o'er research 1uestions or hy#otheses for future research+
K. BI-5;5: 3o the !iases of the authors a'ect the design of the
research or the inter#retation of the results+ -re the authors aware of
#otential !iases and the a'ect on the study+
Grading: for each: great - # ok&poor - $
(,) Issues <listed +y the author=
%hat o#en 1uestions or issues has the author stated remain
unresol$ed+ 3iscuss each in a se#arate #aragra#h of &-1L sentences.
;ach issue2s #aragra#h should ta"e the following format:
what is the issueE
why do you believe this is an important issueE
in what way is it unresolved
suggestions for resolving it 0 if you give your own suggestions instead of or in
addition to the authors'! then precede each with GI would propose ...G If it has been
resolved since the article was written! then state how it was resolved.
Note: If you ha$e any criti1ues in this section, they most li"ely !elong
in the 6eneral <riti1ue section instead.
Grading: 3 items up to > points total ,9 points plus 3 points e.tra
credit/ - for each item: great - 3" ok - #" poor - $
(2) Issues <in your opinion=
Gist se$eral o#en 1uestions or issues which remain unresol$ed in your
o#inion+ >or e.am#le, what #ossi!le future research 1uestions could
arise from this article+ 3iscuss each in a se#arate #aragra#h of &-1L
sentences. ;ach issue2s #aragra#h should ta"e the following format:
what is the issueE
why do you believe this is an important issueE
in what way is it unresolved
suggestions for resolving it
Note: If you ha$e any criti1ues in this section, they most li"ely !elong
in the 6eneral <riti1ue section instead.
Grading: ) items up to $# points total ,9 points plus 9 points e.tra
credit/ - for each item: great - 3" ok - #" poor - $
(4) Impact
To determine how much im#act this article has had, do a citation
analysis. 3iscuss what this citation analysis shows, and whyM don2t 7ust
list the citations8 (5ee the <itation -nalysis 6uidelines (.doc) and
*andout (.#df) #osted on the course %e! site.)
If the article has no citations, then write in your re$iew :I found no
citations in the 5cience <itation Inde., the 5ocial 5ciences <itation
Inde. or on the Internet.: Then clearly e.#lain why you !elie$e there
were no citations at all. If you found citations in some inde.es or on the
Internet !ut not the others, then e.#lain this as well.
Include your citation lists in an a##endi. to your re$iew (see !elow for
details).
Grading - impact discussion: great - 3" ok - #" poor - $
(5) >uestions
Gist three insightful 1uestions of your own, arising from this article. 3o
not as" de,nitions, !ut rather 1uestions that really ma"e one thin".
Grading: 3 7uestions, up to 9 points total - for each 7uestion: great&ok -
#" poor - $
(6) "nnotated *i+liography
>or e$ery item you ha$e cited in your re#ort, you need a full reference
and an annotation e.#laining it. This includes references to any class
materials, as well as the three additional citations utilized in sections 6-
14.
1. Hist the full bibliographic references authors! title! 7ournal name! volume! issue!
year! page numbers! etc." for anything you have cited in your review.
IM5>&+9,+; +his is NOT the bibliography listed at the end of the article. It is
the bibliographic references for any readings you yourself referred to inside your
review.
2. #rite 20) sentences describing the article.
%. #rite 20% sentences describing why you cited it.
-lso, !e sure that you ha$e included a !i!liogra#hic mar"er to each
(such as CBie!er N 5mith, 9LL1E) in the te.t of your re$iew.
Grading: -+ if missing references" -3 if you mention the authors
e.plicitly in your te.t and put the references in this bibliography
section, but forget to e.plicitly place citation markers in your te.t.
(7) Citation "nalysis "ppendi?
There is a se#arate #age on the course %e! site descri!ing citation
analysis.
This a##endi. will ha$e three sections:
the citations you found in the 6cience ?itation Inde1
the citations you found in the 6ocial 6ciences ?itation Inde1
the citations you found through a thorough #eb search on the Internet
If the article has no citations for any of these three, then write in that
section :I found no citations in the C5cience <itation Inde. or the 5ocial
5ciences <itation Inde. or on the InternetE.:
Note, if your article has more than 9L citations, you only need to
include a selection of them:
6tate how many citations each inde1 has and the #eb search found
Hist 102 citations for each year in which the article has been cited. +ry to include
citations from several different 7ournals spread over your selection E Include a
citation analysis to see who has cited it and how.
Grading - citation analysis: # points for each of the regular inde.es,
and # points for the -nternet search" if the article is from $>>* or later
then: $ point for each of the regular inde.es, ) points for the -nternet
search.