You are on page 1of 2

Writing a Review – The Big 9 to Answer

1. Read the introduction/background/results/conclusion and summarise what you think the
paper is about based on those sections in one paragraph.
important because? you need to get a basic idea of the paper, and you need to be able to
validate that what is being claimed is also what will be presented/solved in the paper.
is the research idea original/novel?


is this idea
appropriate for
this conference?

remember: authors should be clear about this!

what is/are the question(s) being
addressed by the author(s)?

is the current
state-of-the-art
well-represented?

2. Validate the claims in the body of the paper. Are the stated research questions addressed? 

Are the authors advancing on the state-of-the-art? hint: best to do this the next day.
important because? the claims need to match up with the actual contents. Do wait a day
before you try to tackle this question. Most papers are evolutionary and come up with
small improvements. At first, you might be underwhelmed by the small contribution and at
the same time feel intimidated as you think you are not an expert. Both are likely not true.
3. Once you have validated that the claim matches the contents, you are ready to judge the
novelty of the paper. Give your score (see below), and then motivate your score in a few
sentences. Use facts to motivate your score, not your emotions!
(7) The paper is highly original, proposing an entirely novel view, idea or problem.
(6) The paper has several novel and surprising contributions.
(5) One idea surprised me with its originality, otherwise solid contributions.
(4) The paper offers worthy contributions, none of which are surprising. (i.e. “next step”)
(3) The paper only offers minor variations to existing ideas.
(2) The paper does not advance our current state of knowledge.
(1) This work is behind on the state-of-the-art, or is a duplicate of existing work.
important because? the novelty of a paper is the single most important criteria to determine if
a paper should be accepted. Don’t expect (all/any?) papers to be revolutionary though!
4. Validate and motivate the technical quality of the paper. You have to check if the results are
technically sound and detailed enough. Are there any obvious flaws? Are the claims wellsupported by either their own novel contributions or related work? Are the results
reproducible? Did the authors discuss both strengths and weaknesses of their approach?
important because? good work is firmly rooted in sound theory and methods. Small ideas that
are well-supported are more useful than big ideas that are subsequently proven wrong.
is the approach
theoretically
sound?

can you think of a better way to
address the paper’s questions?
are relevant papers referenced?

are the
experiments
appropriate?

(5) A solid contribution to a relevant problem. important because? we publish our work for others to read. [optional] Provide comments to the Program Committee (PC) members. I vote for and argue for its acceptance. Give your feedback on the quality of writing. either way is fine by me. At all times motivate your scores and scrutinise the work. and the clarity of the paper overall. (7) A high-impact. Why not give some suggestions to the authors on how they can improve their paper? . (4) A reasonable contribution to a minor problem. (5) A good paper overall that should be accepted. and try to estimate the impact of this work. far reaching (to other domains) contribution. (4) A decent paper overall. (1) Quality of writing is poor and considerable editing is needed. (1) Reject outright.5. this is also about the structure of the paper. Always write in a way that gives the authors the chance to discuss their point of view (you may very well be wrong!). good structure. I vote for acceptance. While syntax and grammar plays an important role. yet could be persuaded otherwise. Always motivate your decision. (1) Educated guess. few typos. Motivate and give your score for the overall quality of this paper. I vote for and argue for rejecting it. I saw something about this during my BSc/MSc studies. but could use some editing. No? important because? knowing your level of expertise helps the Program Committee (PC) members when there are conflicting opinions. Take the novelty and the technical quality into account when doing so. it needs a fair amount of editing. but it need not be a simple weighted average. (4) The quality of writing is excellent. 6. (2) Knowledgeable. a. 8. take all of the previous questions into account. Reject. important because? this single score makes it easy to compare different reviews. To do this. What is your own confidence in your ability to review this work? (3) Expert. not the authors. (3) A weak paper that is just not good enough. Do not be timid about your expertise level. I will fight to get this paper accepted. “Google translation” important because? good ideas buried in an unreadable paper are not very useful at all. This is particularly relevant when you suspect (self-) plagiarism. (3) Unsure how this paper is relevant. (2) Not really a relevant paper for this conference/journal. I will fight to get this paper rejected. or the PC member may ignore or overlook your opinion. (1) Out of scope. (3) The quality of writing is good. clever use of extras. I work on the same topic or have worked in it in the recent past. the good use of figures/tables. when you have concerns about your own bias. not for it to disappear in oblivion.k. (2) The quality of writing is only marginal to ok. (7) An excellent paper. Score and comment on the significance of the paper. (6) A very good paper that should be accepted. 9. this topic was a smaller part of one of my previous works. Accept or reject. Your comments should hold up to the same scientific rigours and should be refutable. and give a reasonable amount of input. (6) Very significant contribution that advances the state-of-the-art. 7.a. (2) A clear rejection. or when you want to rank the papers you have been reviewing but the numbers at your disposal for the scores are inadequate/do not capture the full picture. a very strong accept.