Signature Assignment: Critique of Evaluation Article

Rachael Kowalchick

College of Health and Human Services, California State University Long Beach

HSC 405: Health Education and Measurement

Theodora Papachristou

November 21, 2023

 2

Table of Contents

Description of Study----------------------------------------------------------------------------------- 3

Sampling Method---------------------------------------------------------------------------------------3

Evaluation of Design---------------------------------------------------------------------------------- 5

Internal Validity ---------------------------------------------------------------------------------------6

External Validity---------------------------------------------------------------------------------------8

Conclusion ---------------------------------------------------------------------------------------------9

References----------------------------------------------------------------------------------------------11
 3

Description of Study

The International Journal of Environmental Research and Public Health published “The

Effects of a Skill-based Intervention for Victims of Bullying in Brazil.” the study aims to ‘verify

whether social and emotional skills would reduce victimization among Brazilian 6th-grade

students’ victims of bullying.’ (Silva et al., 2016). An eight-week cognitive-behavioral

intervention is implemented to improve social skills, self-control, emotional expressiveness,

empathy, assertiveness, and interpersonal problem-solving capacity. Data was gathered through

pre- and post-tests; evaluators analyzed data through Poisson regression models with random

effects. Previous programs aimed to reduce bullying through “whole school anti-bullying

programs, curriculum intervention, and social skills training.” (Silva et al., 2016, p.2). This

Brazilian study examined the impact of an intervention for students who were victims of

bullying, incorporating bystander students to create a safe environment to foster social

connections and support, as gathering only victims with similar difficulties would not be

beneficial in learning new skills. The goal was to help victims develop social connections and

receive support from non-aggressive students, establishing a broader social support network

during the intervention.

Sampling

A combination of proportionate Stratified Sampling and convenience sampling

techniques was used for this study. Intervention participants were selected from grade

classrooms within their schools to participate in the program; consent was factored into the final

participant pool evaluators had to choose from, affecting population proportions of subgroups.

As 522 6th graders were invited to participate, only 411 consented, so the judgment/convenient

sampling method could be considered. Due to the need for parental consent and the sampling
 4

place being a school, the method has traces of judgmental sampling bias. Both intervention and

comparison groups were similar in age, ethnic composition, and the number of bystanders and

victims. Participants were not randomly chosen; they were assigned either intervention or

comparison groups within their school.

The sample does not represent the target population due to the non-probability sampling

method of the judgmental/convenience sampling method. The need for parental consent

impacted the study intervention and comparison groups initially, 411 students consented to

participate; however, only 188 students remained involved through the second intervention

phase, with 41.5% of students receiving the intervention and 58.5% in the comparison group, this

exclusion from participation perpetuates inequalities in the research. The absence of control over

participant selection poses a challenge in establishing causal relationships between a lack of

social skills and social networking and the risk of victimization by bullying. The proportionate

stratified random sampling method increases sampling error due to the oversampling in the

comparison group and under sampling in the intervention group. Including bystanders in the

intervention and comparison creates a heterogeneous population, potentially making the sample

less representative of the target population of victims. Although concerning program

implementation, bystanders are vital in creating a developmental habitat for social networking

and skills. A diverse sample with ethnic composition and gender and multiple schools receiving

implementation can be seen as a strength. Each school provides easy access to the sampling

frame and various settings for evaluators to represent victims better as a population. A justified

alternative sampling method could be cluster sampling. From a logistic perspective, 6th-grade

victims are geographically clustered within specific schools. Schools would be considered

clusters, providing easy access and facilitating group intervention due to the nature of the school
 5

environment. Student participants within the same school (cluster) likely have similar

characteristics fostering a comfortable environment for victims. Intra-cluster homogeneity is

accounted for due to the inclusion of bystanders within each cluster, eliminating

overrepresentation or underrepresentation from cluster to cluster. Whereas in stratified sampling,

each stratum should have a homogenous population. The cluster method is a more suitable

method for this study. However, it is essential to consider sampling variability. Cluster

variability could reduce estimates' precision due to wider confidence intervals.

Evaluation of Design

The Effect of a Skill-based intervention for victims of bullying in Brazil closely corresponds

with a Quasi-experimental design nonequivalent comparison group. The study consists of an

experimental group and a comparison group created through a self-reported Aggression and Peer

Victimization (EVAP) scale, allowing evaluators to determine and distinguish the two groups

using the Ward Method. The Ward method “consists of a hierarchical grouping procedure in

which the similarity measure used to group individuals is calculated as the sum of squares

between two groups” (Silva et al., 2016, p. 6). Evaluators created the experimental and

comparison groups as the average proportion of (40-50%) victims and (50-60%) bystanders. Due

to the hand-selected groups based on EVAP results, this program lacks randomization in the

experimental and comparative groups. However, each group was given a pre-test and a post-test.

The pre-test was administered in the first week of March, a month into school, and the post-test

occurred in June, seven days after the program ended. Results were compared through the

Poisson regression model. For the study, these were the best levels of measurement due to the

minimization of internal variation due to the equal sizes of clusters.

 6

Internal Validity

Internal validity is the extent to which the study accurately measures what it is intended to

measure. In the case of this study, the objective is to “verify whether improved social and

emotional skills would reduce victimization.”. One threat to internal validity is the timing of the

peer report. The peer report is on a sociometric scale, excessively ranking their classmates,

whom they would enjoy spending time with, and whom they would stray from. This report was

given at the beginning of the school year. For many students, the beginning of the school year is

a transition period; reactionary behavior due to this change of environment and interactions with

peers may provide false data. However, if it were implemented later in the school year, the

student would have had time to adjust, have a better understanding, and strengthen relationships

with their classmates.

Selection bias may pose another threat to internal validity. Systematic differences between

experimental and control groups at the beginning of the study may influence observed outcomes

concerning the study's success. Additionally, self-selection bias will contribute to threats to

internal validity. Participants who engage in the intervention may have heightened motivation

and be more accepting of the intervention material. Any observed positive outcome may be

attributed to existing distinctions rather than the actual efficacy of the intervention. Another

critical bias to take note of is school and teacher selection bias. School resources vary, as well as

the environment. Although the intervention is only implemented in select groups, they are active

participants in their environment. Relationships with staff who are implementing the program

will affect behavior outcomes. These observed effects may be confounded by these differences

rather than solely attributable to the intervention. To mitigate the threat of selection bias,
 7

evaluators should do their best to carefully control relevant variables through statistical

techniques to minimize the impact of selection bias on internal validity.

Additionally, historical threats, external events, or changes in the participating students’ school

environment could influence outcomes. Policy changes in schools may alter the environment that

participants engage in. Concurrent anti-bullying awareness campaigns implemented at a school-

wide level; victims' behavior may be influenced by the broader school-wide initiative rather than

the specific skill-based intervention. To control the historical threat, evaluators can include a

control group and be aware of external factors. Monitoring and documenting all possible events

can assist in interpreting study results.

Maturation can also be a threat to internal validity. Student participants will naturally mature

throughout the implementation of the study; this maturation is unrelated to the intervention but

can still affect intervention outcomes. For example, a sixth grader may develop their coping

skills and resilience over the school year. The natural development of coping skills and resilience

could be mistaken for intervention effects. Decreasing the program duration will decrease the

impact of maturation on participants.

Another possible threat to internal validity may be regression to the mean. Extreme scores on the

pretest will inherently move closer to the mean on subsequent measurements. The mature level

of sixth graders and their willingness to participate will dictate the results. For example, some

participants may complete the survey without reading the questions. These results can be

mistaken for an intervention effect. The intervention group can be compared to the control group

to mitigate regression to the mean. Depending on the improvements made by the intervention

group, if the improvement is beyond what could be expected based on regression to the mean,

the argument for the intervention's effectiveness is strengthened.

 8

Testing effects are likely to be a threat to internal validity. Students may change their behavior

because they know the intervention or are being observed. Pre and post-test results can be

inaccurate because students will likely recall the assessment. A placebo and an ‘alternative’

intervention can be implemented in the comparison group to control testing effects.

Interaction primarily affects internal validity, especially within the middle school student

population. Implementation of the program is presented by school staff. Students’ participation

may depend on their relationship with the instructor and vice versa. Relationships external to the

program will still influence results through over-participation or under-participation. This can be

controlled by bringing in a third party without previous relationships with the participants.

External Validity

External Validity is concerned with the generalizability of the study's results to the broader

context of bullying victimization. One threat to external validity could be population validity;

due to the program being implemented across various schools, the study does not account for the

unique characteristics of each school’s population. Victims, as a general population, may not

have the same struggles as those who are bullied outside a school setting.

A major threat to external validity in this study is Ecological validity. The study conditions were

considered ‘artificial’ and, therefore, not representative of the conditions of a real-life bullying

scenario. Results may report improved behavior and social skills, but due to the classroom

conditions of the implemented program, results are invalid when the scenario is applied. With

behavior, the Hawthorne effect threatens external validity; the Hawthorne effect is the possibility

that participants may alter their behavior because they are aware they are being observed. These

findings may not accurately reflect their typical behavior in actual world situations.
 9

Due to selection bias, the study’s external validity was threatened by the fact that the sampling

method was technically a convenient sample. The consent factor of participation is necessary;

however, it limited the accurate representation of the sample population. The population may not

accurately include specific demographic groups, socioeconomic status, or cultural background.

Therefore, findings may not be generalizable to populations, limiting the external validity across

different population segments that chose to withdraw from the study.

It is essential to address the threat of demoralization. The study aims to improve the social skills

of participants who are victims of bullying to reduce bullying and harassment. Considering

human nature, it is crucial to address the participant dropout rate or inaccurate data due to the

lack of motivation and morale of those who are bullied regularly. Demoralized participants may

have altered response patterns to interventions and measures compared to participants who are

not demoralized. Due to fluctuating response and effort patterns, demoralization may not reflect

how the intervention will be received in a more general or diverse population.

Conclusion

The effects of a Skill-based Intervention for Victims of Bullying in Brazil address middle school-

level bullying by implementing an intervention that focuses on building social skills for the

victim. This study’s objective was to ‘verify whether or not improved social and emotional skills

would reduce victimization among Brazilian students who were victims of bullying attending the

6th grade’ (Silva et al., 2016, pg.3.) Directly addressing victims was a unique approach as seen

school-wide intervention have had the most success in the past. It can be inferred that sampling

bias occurred due to the participant dropout rate; the article did not clearly state a specific

sampling method, although characteristics deduce traces of judgmental sampling. They were

leading the experimental and control groups to lack randomization. Controlling dropout rates and
 10

intervention participation would help evaluators select the appropriate sampling method and

simultaneously increase the accuracy of results. The chosen design of the study was a pretest and

post-test. Alternative designs would be advised as there were many threats to internal validity,

leading to the possibility of inaccurate results. Threats to external validity are stated towards the

end of the article, addressing the weakness and lack of generalizability. In conclusion, a

curriculum-based school-wide intervention would be advised. School-wide intervention allowed

evaluators to establish specific sampling methods, guaranteeing randomization within the

intervention school. Another school similar in socioeconomic status, demographically, and

population-wise, should be selected for the control school. With proper planning and

implementation, all students can benefit from improved social skills and awareness,

strengthening the student body and creating an environment that does not tolerate bullying.
 11

References

Silva, J., Olivieria, W., Braga, I., Farias, M., Lizzi,E., Goncalves, M., Pereira, B., & Silva, M.

(2016). The Effects of Skill-Based Intervention for Victims of Bullying Brazil