RESEARCH METHODOLOGY

Evaluating research paper

M19511_FARA
Article 1: Music training improves speech-in-noise perception: Longitudinal evidence from a
community-based music program.
 Ch 2. Evaluating Titles
Questions
1. Is the title sufficiently specific?
2.Is the title reasonably concise?
3.Are the primary variables referred to in the title?
4. When there are many variables, are the types of
variables referred to?
5.Does the title identify the types of individuals
who participated?
6. If a study is strongly tied to a theory, is the name
of the specific theory mentioned in the title?
7. Has the author avoided describing results int he
title?
8. Has the author avoided using a “yes–no”
question as a title?
9. If there are a main title and a subtitle, do both
provide important information
about the research?
10.If the title implies causality, does the method of
research justify it?
11.Is the title free of jargon and acronyms that
might be unknown to the audience for
the research report?
12. Are any highly unique or very important
Score Evidence
5 The title are sufficiently specific enough to
introduce the main content of the study articles.
Also, form the title it is safe to assume that the the
articles can be differentiated from other articles on
the same topic and sufficient enough for the
reader to make a judgement relevant enough to
their needs.
4 The title contained 15 words that are specific
enough for the researcher to locate the articles but
not overly informed.
3 Two variables were described here which are
music training and speech in noise perception
N/A In this study, researcher already specified that the
outcome that being observed are the speech-in-
noise perception improvement of the participant
after the community-based music program.
N/A The author did not included the name of
population that being included as the participant
(population of interest).
The focus group also are not described in the title.
However, in this articles, it is specified that the
population of interest are children. In my personal
opinion, title should include that population of
interest as there are huge difference between adult
and children in term of music training efficacy
especially to their performance of speech in noise
perception.
N/A This study did not strongly related to any theory.
Thus, it is not applicable to evaluate the question.
1 The author described how music training can
improves the speech in noise perception and also
describe how this can be proved by community-
based music program.
N/A Author did used yes-no questions as the title.
4 Author use appropriate two part title. Both part
provide important information
First part: music training and speech-in noise
perception.
Two: usage of community-based music program
(method)
N/A Author did not use any 'effect' words or any
synonym to 'effect'. even though the main
outcome is to see to effect of music training to the
speech-in-noise perception.
5 There is no acronym or jargon used in titles.
4 The title mentioned about music training but
 Evaluating research paper
Article 2: The perception of prosody and associated auditory cues in early -implanted
children: The role of auditory working memory and musical activities.
M19511_Fara
Ch 2. Evaluating Titles
Questions Score Evidence
5 The title are specific can be differentiated from the other
1. Is the title sufficiently specific? research articles of the same topic

2. Is the title reasonably concise? 2 The title consist more than 15 words which are consider as
quite long and provided more information.

5 All three variables were stated in the title

3. Are the primary variables referred to in the i) perception of prosody
title? ii) early-implanted children
iii) music training/ auditory working

4. When there are many variables, are the types of
variables referred to?
5. .Does the title identify the types of individuals
who participated?
3 The associated auditory cues/ musical activities here consist of
three different type (author just stated about musical
activities).
5 The type of individual here were stated as early- implanted
children.

6. If a study is strongly tied to a theory, is the N/A The research did not closely related to specific theory.
name of the specific theory mentioned in the title?
7. Has the author avoided describing results int he 5 Author did not stated any results in the title.
title?
8. Has the author avoided using a “yes–no” 5 Author did not stated any 'yes-no' question in the title
question as a title?

9. If there are a main title and a subtitle, do both 5 Both main title and a subtitle are both refer to specific
provide important information variables examined in research study.
about the research?

N/A The author did not implies any causal relationship directly by
using the word 'effect' but instead investigating the roles of
10. If the title implies causality, does the method
auditory working memory and musical activities in the
of research justify it?
improvement of perception of prosody. (comparison between
CI user n normal hearing).

11. Is the title free of jargon and acronyms that 5 Title are free of jargon and acronyms that might be unknown
might be unknown to the audience for .
the research report?

5 Yes, the characteristic; not only children CI user but also early
12. Are any highly unique or very important
implanted CI children.
characteristics of the study referred to in the title?
4 Title are clear, well explained, and the important information
13. Overall, is the title effective and appropriate? were stated in the title.

Ch 3. Evaluating Abstracts
Objective: To study prosodic perception in early-implanted children in relation to auditory discrimination, auditory working
memory, and exposure to music. Design: Word and sentence stress perception, discrimination of fundamental frequency (F0),
intensity and duration, and forward digit span were measured twice over approximately 16 months. Musical activities were
assessed by questionnaire. Study sample: Twenty-one early-implanted and age-matched normal-hearing (NH) children (4 – 13
years). Results: Children with cochlear implants (CIs) exposed to music performed better than others in stress perception and
F0 discrimination. Only this subgroup of implanted children improved with age in word stress perception, intensity
discrimination, and improved over time in digit span. Prosodic perception, F0 discrimination and forward digit span in
implanted children exposed to music was equivalent to the NH group, but other implanted children performed more poorly.
For children with CIs, word stress perception was linked to digit span and intensity discrimination: sentence stress perception
was additionally linked to F0 discrimination. Conclusions: Prosodic perception in children with CIs is linked to auditory
working memory and aspects of auditory discrimination. Engagement in music was linked to better performance across a
range of measures, suggesting that music is a valuable tool in the rehabilitation of implanted children.

Questions Score Evidence

1. Is the purpose of the study referred to or at least 5 The purpose of study were clearlly stated and implied in the
clearly implied? objective section of the abstract (structured abstract)

the design and methodology of the articles was appropriately

2. Does the abstract mention highlights of the
describe in the Design section of the abstract (Structured
research methodology? 4
abstract)

3. Has the researcher omitted the titles of even though the main musical activities was assessed with
measures (except when these are the focus questionnaire, author avoided stating the full name of
5
of the research)? questionnaire.

The highlight was stated; for children qith CIs, word stress
perception was linked to digit span and intensity
4. Are the highlights of the results described?
4 discrimination: sentence stress perception was additionally
linked to F0 discrimination.

5. If the study is strongly tied to a theory, is the N/A Study was not strongly related to the theory.
theory mentioned in the abstract?

6. Has the researcher avoided making vague Author appropriately described the implication of their
references to implications and future research without vaguely described them.
5
research directions?

The abstract content; appropriately described; without vaguely

7. Overall, is the abstract effective and
over stated their finding;
appropriate? 4
Further, the information were written in short and clear ways.

Ch 4. Evaluating Introductions and Literature Reviews

Questions Score Evidence
Author describe the population in focus and the limitation of
1. Does the researcher begin by identifying a
that the population is having.
specific problem area? 4
(appropriately board description)

The author specifically address the problem and the reasons of

the problem (focus) This often leads to poorer or more
2. Does the researcher establish the importance of variable speech and language skills in children with CIs than
5
the problem area? in children with normal hearing (Geers et al, 2003;
Pisoni et al, 2011).

3. Are any underlying theories adequately N/A Not strongly related to the theory.
described?

4. Does the introduction move from topic to topic The introduction was organized around topic and the
instead of from citation to references site together instead of series of references being
4
citation? sites other after another.

5. Are very long introductions broken into The introduction were considered as long introduction
subsections, each with its own however, author did not use any subheading.
2
subheading?

Author did not provide formal conceptual definitions. (Might

6. Has the researcher provided adequate
not be needed as most of the words definition already well
conceptual definitions of key terms? I/I
known)

7. Has the researcher cited sources for “factual” Each statement/ fact were cited from source (past study/
statements? 5 book).
8. Do the specific research purposes, questions, or
hypotheses logically flow from
the introductory material?
9. Overall, is the introduction effective and
appropriate?
Ch 5. A Closer Look at Evaluating Literature Reviews
Questions
1. Has the researcher avoided citing a large
number of sources for a single point?
2. Is the literature review critical?
3. Is current research cited?
4. Has the researcher distinguished between
opinions and research findings?
5. Has the researcher noted any gaps in the
literature?
6. Has the researcher interpreted research
literature in light of the inherent limits of
empirical research?
7. Has the researcher avoided overuse of direct
quotations from the literature?
8. Overall, is the literature review portion of the
introduction appropriate?
Ch 6. Evaluating Samples When Researchers Generalize
Questions
1.Was random sampling used?
2. If random sampling was used, was it stratified?
3. If some potential participants refused to
participate, was the rate of participation
reasonably high?
4. If the response rate was low, did the researcher
make multiple attempts to contact potential
participants?
5. Is there reason to believe that the participants
and nonparticipants are similar on relevant
variables?
6. If a sample is not random, was it at least drawn
from the target group for the
generalization?
All the specific research purpose, question were stated in the
last
5
Overall, the introduction content can be consider as
4 appropriate. Each statement that was included were cited from
reliable sources.
Score Evidence
The author break the citation into subgroup according to the
population and also the factor. E.g Despite the importance of
prosodic stress for the segmentation of speech, there are only a
handful of studies assessing prosodic stress perception in
4 children with CIs (Word and sentence stress: Klieve & Jeanes,
2001; Most & Peled, 2007; O ’ Halpin, 2010. Word stress:
Lyxell et al, 2009, Titterington et al, 2006. Imitation of word
stress patterns: Carter et al, 2002).
The author provide both positive and negative critics to the
4 past paper.
the references that been cited are from the latest publication
5 (not more than 10 years old).
Author did not used wording that can point out if the cited
literature present opinions or research result.
2
the knowledge gaps was clearly explained several times in the
4 review.
N/A
N/A
author did not use any direct quotation from any references.
5
The introduction of this study adequately mentioned the
results and the author's opinions from the background to the
O
purpose and hypothesis of the study and adequately described
the introduction of this study.
Score Evidence
The randomization of sample was not clearly stated.
I/I
Not clearly stated.
I/I
Not clearly stated. However, as the focus group of the study
was children with cochlear implant, in personal opinions it
I/I
was reasonably adequate.
N/A
N/A
N/A
N/A
In this study, it can be conclude that convenience sample was
used. All the participant in this study were the participant from
5
previous study conducted by the same researcher.
7. If a sample is not random, was it drawn from N/A
diverse sources? N/A

8. If a sample is not random, does the researcher The limitation of the study regarding the subject selection was
explicitly discuss this limitation? 1 not properly discuss in this study.

9. Has the author described relevant demographics The demographic of subject was adequately stated and
of the sample? 5 summarized in the table.

From personal opinion, the sample size are consider as

adequate, how ever, the description of subject, and the
10. Is the overall size of the sample adequate? limitation in generalizing the subject are really limited and not
3
enough information was stated (need to be careful in
generalizing the result to the focus population).

The participant was divided into two subgroup (musically

11. Is the number of participants in each subgroup active vs musically inactive group). However, no clear
sufficiently large? 2 description how many subject been assigned to each group
(Only one group number participant was mentioned)

informed content was given to t he parent prior to testing and

12. Has informed consent been obtained? for the participant, the consent were gave orally (after study
5
was explained to them).

Even thought the number of subject was considered as

adequate, However, in literal reading, author does not provide
clear information on the requirement process. Author also did
not describe some of the important information that limit the
13. Overall, is the sample appropriate for
1 reader to evaluate whether appropriate subject was requited.
generalizing?
Furthermore, some of the information was not included
directly in the paper. instead, the reader need to find it at the
online website of the journal.

Ch 7. Evaluating Samples When Researchers Do Not Generalize

Questions Score Evidence
some of the information was not included directly in the paper.
1. Has the researcher described the
instead, the reader need to find it at the online website of the
sample/population in sufficient detail? 3
journal. (might be conveniece for the reader).
.
2. For a pilot study or developmental test of a Not a pilot study
theory, has the researcher used a sample with
N/A
relevant demographics?

3. Even if the purpose is not to generalize to a Sample size are consider as adequate especially for the focus
population, has the researcher used a group that are limited (children with cochlear implant).
3
sample of adequate size?

4. Is the sample size adequate in terms of its N/A

orientation (quantitative versus
N/A
qualitative)?

5. If a purposive sample has been used, has the The inclusion criteria of subject was clearly stated in the
researcher indicated the basis for method section.
3
selecting participants?

N/A
6. If a population has been studied, has it been
clearly identified and described? N/A

N/A
7. Has informed consent been obtained?
N/A

8. Overall, is the description of the sample
adequate?
Ch 8. Evaluating Instrumentation
Questions
1. Have the actual items and questions (or at least
a sample of them) been provided?
2. Are any specialized response formats, settings,
and/or restrictions described in
detail?
3. When appropriate, were multiple methods used
to collect data/information on each variable?
4. For published measures, have sources been
cited where additional information
can be obtained?
5. When delving into sensitive matters, is there
reason to believe that accurate data
were obtained?
6. Have steps been taken to keep the measures
from influencing any overt behaviors
that were observed?
7. If the collection and coding of observations
involves subjectivity, is there evidence of
interobserver reliability?
8. If a measure is designed to measure a single
unitary trait, does it have adequate internal
consistency?
9. For stable traits, is there evidence of temporal
stability?
10. When appropriate, is there evidence of content
validity?
11. When appropriate, is there evidence of
empirical validity?
12. Do the researchers discuss obvious limitations
of their measures?
13.Overall, are the measures adequate?
Ch 9. Evaluating Experimental Procedures
Questions
1. If two or more groups were compared, were the
participants assigned at random
to the groups?
This is are quantitative study, thus, the evaluation of sample in
qualitative study might not be appropriate.
5
O/X Evidence
This study measured digit span and also PIQ of the participant.
For PIQ, it was measured with the block design subtest of the
5 WISC-IV (Wechsler intelligence scale for children, 4th
edition: Wechsler, 2010).
The outcome was measured in setting that are specific for
measuring hearing ability (Soundproof room), and also the
response also was recorded by specific test (ITPA/PIQ).
4
Program, volume, and sensitivity level were
adjusted to the clinically recommended values.
The performance of subject was measured in different
5 measure (Sentences stress, word stress, Fundamental
frequency perception, digit span, PIQ)
The software that been used for data analysis was SPSS. All
the additional information of how they were analyse and the
4
sources have been cited.
N/A
N/A
N/A
N/A
All sounds were presented using the everyday settings of the
CI, without use of any acoustic hearing aid. Program, volume,
4 and sensitivity level were adjusted to the clinically
recommended values.
N/A
N/A
N/A
N/A
The main statistical analysis was done by using linear mixed
modelling that have been propose by Singer & Wilett, 2003;
O West, 2009). How LMM works were also described in the
methodological sections.
N/A
N/A
N/A
N/A
This study conducted a correlation and LMN analysis to
identify the association between the music training and the
O
behavioral response. All the test and measure procedure were
justified and clearly stated in the paragraph.
O/X Evidence
5 The research compared two group which are CI group and
also the normal group. Participant of CI group also was
grouped into two subgroup.
2. If two or more comparison groups were not
formed at random, is there evidence
that they were initially equal in important ways?
3. If only a single participant or a single group is
used, have the treatments been
alternated?
4. Are the treatments described in sufficient
detail?
5. If the treatments were administered by
individuals other than the researcher, were
those individuals properly trained?
6. If the treatments were administered by
individuals other than the researcher, were
they monitored?
7. If each treatment group had a different person
administering a treatment, did the
researcher try to eliminate the personal effect?
8. If treatments were self-administered, did the
researcher check on treatment
compliance?
9. Except for differences in the treatments, were
all other conditions the same in the
experimental and control groups?
10. When appropriate, have the researchers
considered possible demand
characteristics?
11. Is the setting for the experiment natural?
12. Has the researcher distinguished between
random selection and random assignment?
13. Has the researcher considered attrition?
14. Has the researcher used ethical and politically
acceptable treatments?
15. Overall, was the experiment properly
conducted?
Ch 10. Evaluating Analysis and Results Sections: Quantitative Research
Questions
1. When percentages are reported, are the
underlying numbers of cases also
reported?
2. Are means reported only for approximately
The randomization was not stated. And also, the subgroup was
divided by using inclusion criteria for each subgroup.
1
N/A
N/A
The music training and all the procedure that have been
O conducted was clearly stated. The setting of training, duration
and also the stimuli used in the music training were described.
In this research, researcher conduct the study by themselves
and the participant parent (home based musical activities). The
4
instructions were clearly explained to the parent before the
experiment begins.
As the training was conducted at home and the nature of
training program are home-based music training, the parents
2
was not monitored. The data was collected form parent by
using questionnaire.
As the setting of musical training was done both at center and
also home, the personal effect might been eliminated (home
N/A setting- parent/participant interaction) However when the
training conducted at center, this effect might not have been
eliminated.
N/A
N/A
The NH group was matched to the CI group by age, gender,
and musical activities
5
N/A
N/A
This study conducted an analysis of the perception of speech
in noise after music training for each study subject. All the
setting of data collection and only the setting of center was
describe (limited to describe the home setting). However, in
2
term if external validity, as most of training was conducted at
home and the data collection was conducted in clinical setting
and sound proof room, the external validity in this research
might be low
The author did not state if the selection was made random or
not. However, it is clear that author did not followed random
1
assignment to treatment condition (followed inclusion/
exclusion criteria).
N/A
N/A
N/A
N/A
The study was carried out in accordance with the Declaration
of Helsinki, and all procedures were approved by the local
3
ethical committees of the participating hospitals.
Score Evidence
N/A
N/A
O In this study, the statistical significance and also the
symmetrical distributions? interaction of each theoretical variable was conducted through
regression, ANOVA, and LMM.
3. If any differences are statistically significant The author described some of the similar/ not big differences
and small, have the researchers in the outcome between group; Duration discrimination was
5 broadly similar across the CIm, CIn, and NH groups.
noted that they are small?

Yes, the result section was also described in essay; the result
4. Is the Results section a cohesive essay?
5 were describe in the paragraph, each describes the aspect of
the result.
5. Does the researcher refer back to the research All the hypothesis, and questions that have been stated in the
hypotheses, purposes, or questions introduction paragraph was then described in the result part.
O
originally stated in the introduction?

6. When there are a number of related statistics, As stated, study used linear mixed modeling (LMM) for the
have they been presented in a data analysis. All the value, regression and outcome data was
5
table? summary in the Result table.

7. If there are tables, are their highlights discussed Each statistical data estimated is clearly stated in the results of
in the narrative of the Results section? 5 this study

8. Have the researchers presented descriptive The descriptive analysis here were measures of variability
statistics before presenting the results of (SD) and correlation coefficient was used to describe the
5
inferential tests? direction and strength of relationship between the variables.

9. Overall, is the presentation of the results To test the purpose and hypotheses of the study, various and
comprehensible? sophisticated statistical techniques were analyzed and the
techniques of the results were systematically reported in table
5
10. Overall, is the presentation of the results and narratively explained in the paragraphs.
adequate?

Ch 11. Evaluating Analysis and Results Sections: Qualitative Research

Questions Score Evidence
The study is quantitative study. (Non applicable)
Ch 12. Evaluating Discussion Section
Questions Score Evidence
1. In long articles, do the researchers briefly The main focus of research was explained by using the
summarize the purpose and results at the subheading (clearly stated the hypothesis/research question
4
beginning of the Discussion section? one by one)

2. Do the researchers acknowledge specific No limitation of methodological limitation was described.

methodological limitations? 1

3. Are the results discussed in terms of the Most of the issue cited from past study at introduction was
literature cited in the introduction? address and discussed in the discussion paragraph.
5
4. Have the researchers avoided citing new
references in the Discussion section?

Yes, the author described the specific implication were

discussed; These links suggest that auditory discrimination
5. Are specific implications discussed? and auditory working memory should be important targets in
5
the rehabilitation of early implanted children, in order to
improve perception of word and sentence stress.

6. Are the results discussed in terms of any N/A

relevant theories? N/A

Author suggested that an experimental study would be the only way

to test the hypothesis that musical activities from an early age can
7. Are suggestions for future research specific?
5 enhance these and perhaps also other important aspects of speech
and language.

8. Overall, is the Discussion section effective and 4 The consideration of this study clearly presents the test results
appropriate? for the purpose and hypothesis of the study mentioned in the
introduction.