International Journal of Children's Spirituality

ISSN: 1364-436X (Print) 1469-8455 (Online) Journal homepage: http://www.tandfonline.com/loi/cijc20

Towards a model for evaluating spiritual

recollections

Jose Eos Trinidad

To cite this article: Jose Eos Trinidad (2018): Towards a model for evaluating spiritual
recollections, International Journal of Children's Spirituality, DOI: 10.1080/1364436X.2018.1456411

To link to this article: https://doi.org/10.1080/1364436X.2018.1456411

Published online: 26 Mar 2018.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=cijc20
International Journal of Children’s Spirituality, 2018
https://doi.org/10.1080/1364436X.2018.1456411

Towards a model for evaluating spiritual recollections

Jose Eos Trinidad 
Department of Interdisciplinary Studies/Institute for the Science and Art of Learning and Teaching,
Ateneo de Manila University, Quezon City, Philippines

ABSTRACT ARTICLE HISTORY

Spiritual recollections are often evaluated through the Received 12 December 2017
effectiveness of delivery, rather than the outcomes of the Accepted 20 March 2018
recollections. Such preference is understandable given the KEYWORDS
difficulty in defining and quantifying recollection outcomes. Catholic university students;
To address this gap, the article proposes a framework for multifaith; quantitative
evaluating recollection outcomes rather than delivery, analysis; program evaluation
and shows how a Filipino Catholic university makes use
of this framework for its spiritual recollections. Using data
from the university’s Office of Campus Ministry, the study
shows how these outcomes were defined and finds that
prayer significantly increases the odds of gaining from the
recollection while participation in the alternative multifaith
group reduces these odds. The article details the changes
that were made in response to these results and ends with
possible adaptation of this model in other contexts.

Introduction
The rise of school accountability measures has led to educational programmes
and interventions being constantly evaluated for its efficacy and impact (Hult
and Edström 2016; Ingersoll 2003; Ravitch 2010). Feeding programmes in devel-
oping countries are evaluated if they increase school enrolment and attendance
(Jayaraman and Simroth 2015). Preschool interventions are assessed for its impact
on long-term adult outcomes (Garces, Thomas, and Currie 2002; Heckman, Pinto,
and Savelyev 2013). Charter schools are also judged if they have significantly higher
test scores when compared to neighbourhood public schools (Angrist, Pathak,
and Walters 2013; Dobbie and Fryer 2011). As educational systems continue to
measure different programme’s impact, the same tendency for evaluation does
not evade school programmes that address the spiritual needs of students. As
half-day activities that integrate prayer points and reflection, spiritual recollections
also need to be evaluated (Hatch et al. 1998). With this trend, how can spiritual
recollections be evaluated?

CONTACT  Jose Eos Trinidad  jtrinidad@ateneo.edu

© 2018 Informa UK Limited, trading as Taylor & Francis Group
2   J. E. TRINIDAD

In spiritual recollections, the most common evaluation question is how effective

was the delivery of the recollection. Programmes use a Likert scale with people
agreeing or disagreeing to statements such as: ‘The speaker delivered the pro-
gram well. The venue was appropriate for the program. There was enough time
for the recollection’. Most of the questions centre on the way the programme
was conveyed and changes are made based on what factors received low marks.
Concentrating on the delivery, however, does not provide a sense of the impact
of the programme. One does not know if the recollection led to higher grades,
improved moral living, or closer relationship with God. Thus, the challenge is to
search for variables that can approximate the impact of recollections.
In this article, recollection data from a Filipino Catholic university are used. The
data do not only contain evaluation items for the programme’s delivery but also
include items that can approximate the desired outcomes of the recollections. With
this data, the Office of Campus Ministry (OCM) is able to see if they have achieved
their target and understand what factors contribute to students benefiting from
the recollection. This article outlines the process for determining effectiveness and
quantifying predictors. It also aims to diversify the evaluation techniques used
for spiritual activities, and contribute to better understanding the factors that
influence the programme’s impact.

Program evaluations literature

More sophisticated tools for analysing data and a greater need for impact assess-
ment are two important factors that led to the rise of different forms of evaluations
(Unterhalter 2017; Vanclay 2003). Social and behavioural scientists and manage-
ment researchers have closely scrutinised programmes, products, events and inter-
ventions. For example, economists study labour programmes and government
policies for their impact on beneficiaries (Angrist and Krueger 1999; Hamermesh
2017). In schools, interventions are closely monitored to see the impact on students
and their learning (Durlak et al. 2011; Germán et al. 2017). With this demand for
data, information and evaluation, researchers have created different models to
assess satisfaction with and impact of particular programmes.
More than an academic practice, program evaluation is generally understood as
a practical investigation to understand the success of a programme and provide
information that can improve its implementation (Royse, Thyer, and Padgett 2015).
In the field of education research, the Kirkpatrick evaluation model is used to com-
prehend the effectiveness of a training or educational intervention (Kirkpatrick and
Kirkpatrick 2016). The model has four levels: reaction, learning, behaviour and results.
The first level measures the satisfaction of training participants; the second evalu-
ates the level of knowledge or skills developed from the training; the third looks at
behavioural changes stemming from it; and the last assesses more long-term effects
on organisational impact (Paull, Whitsed, and Girardi 2016; Reio et al. 2017). These
four levels indicate that evaluation can come in different forms, and evaluation’s
purposes are just as varied – spanning the practical–theoretical continuum.
 INTERNATIONAL JOURNAL OF CHILDREN’S SPIRITUALITY   3

From a practical standpoint, evaluations help justify funding for a programme,

show how improvements can be done, evaluate impact of new interventions, and
provide measures to ensure accountability. One purpose is for scarce resources in
education interventions to be channelled to programmes that are actually effec-
tive, ensuring that money is spent well (Emshoff 2008; Laudel 2006). A second
rationale for programme evaluation is to see how programmes can be improved
and changed, what aspects to strengthen and what populations to reach out to
(Beare et al. 2012). A third is the need to evaluate how interventions impact the par-
ticipants, especially when these interventions could actually have negative rather
than positive consequences (Geven, Skopek, and Triventi 2017; Pinfold et al. 2003).
From a theoretical standpoint, there are program evaluations that could actu-
ally drive or be driven by theory. Using theory-driven evaluation helps research-
ers learn not only if the programme was effective but also what key constructs,
structures or logic influence the effectiveness of programmes (McDavid, Huse, and
Hawthorn 2012). In this way, the empirical evidence obtained from evaluations
can help shape or strengthen theory (Petrosino 2000). When a researcher has a
theory, quantitative evaluations are particularly helpful in testing how variables
correspond or model the theory. It is here that something as practical as program
evaluation could have a more theoretical purpose(Brock, Durlauf, and West 2007;
Fitz-Gibbon and Morris 1996).
Given the many practical and theoretical uses of program evaluations, these
evaluations take on many forms as well (Rossi, Lipsey, and Freeman 2004). One form
of program evaluation is assessing the needs for a programme, where it examines
the feasibility of the programme, the possible reception and how it can have an
impact (Grant 2002; Korth 2001). Another type of evaluation is the assessment
of people’s satisfaction with a programme, where participants are asked about
their contentment with the programme’s implementation and what could be
improved (Yamashita et al. 2015). There are evaluations to assess the efficiency of
programmes, that is, if the costs are healthily balanced by the programme’s bene-
fits (Batton 2003; Block Joy, Goldman, and Pradhan 2006; Stoneberg 2015). Finally,
evaluations assess the impact of programmes (Castiglia and Turi 2011; Marcella,
Lockerbie, and Bloice 2016; Ngware et al. 2015).
In impact evaluations, there is a target change or outcome, and it is investigated
if the change has been significant enough since the interventions started (Coe,
Arthur, and Hedges 2017). It must be noted that the change should be a change
in the target population (e.g., behaviours, grades or learning) or social conditions
(e.g., less dropouts, higher college-going rates) – and that these changes should
be both empirically significant and theoretically attributable to the intervention
(McDavid, Huse, and Hawthorn 2012).
With these different forms of evaluations and assessments, it is important to
understand how spiritual activities – specifically those in schools – are evaluated.
One example is Belmont University’s spiritual growth and discernment ques-
tionnaire, where students answer qualitative responses that show their spiritual
4   J. E. TRINIDAD

development (Schneller, Minardi, and Lake 2016). Students who spent a summer
term in Haiti answered the questionnaire, and most reported positive growth in
cultural awareness, respect for identities and articulation of personal faith and
spirituality. Here, evaluation takes the form of the change in the person’s worldview
similar to the assertion of Astin, Astin, and Lindholm (2011) that colleges could help
facilitate students’ spiritual development. This qualitative understanding of growth
through a questionnaire is one way of evaluating spiritual activities in schools.
Another way of evaluation is through quantitative means where people are
asked primarily about their satisfaction with a programme. For example, a hospital
chaplaincy asked respondents if they felt welcomed, safe, connected with God and
connected with others (Tiano and Maclean 2015). In this type of satisfaction evalu-
ation, it usually considers the percentage of people who respond positively to the
spiritual programme. Another way of quantitatively analysing programmes is using
satisfaction as the outcome variable, and predictors such as the dispositional hope
scale and personality traits. Thus, this process uses a more predictive method for
understanding what factors contribute to satisfaction in the spiritual programme.
In this current study, rather than use satisfaction, the outcome variable is impact.
Similarly, rather than use personality or disposition, the predictor variables are
the student’s participation and engagement in the recollections. By providing this
alternative model for assessing spiritual recollections, organisers can learn what
factors help people benefit from recollections.

Spiritual recollection data

Year-level recollections
The present study’s locus is a Catholic university in the Philippines, here called
Marian University. It is a fairly large university, run by a religious order of the Roman
Catholic Church. Although run by religious, its faculty and staff are mostly lay
people. Particularly, the OCM is mainly composed of lay people with priest-chap-
lains who come on a part-time basis. There are four campus ministers devoted
exclusively for each of the four year levels.
A student in the Marian University will usually have one spiritual recollection
activity for every year. Freshman students have their recollection as part of the
university’s introductory course to its school culture, a course that all freshmen
take. In this recollection, students are focus on the sense of gratitude that one feels
for God’s gifts. Sophomores have their recollections as part of the national service
course, where students go to underserved communities and work on literacy or
community development. One Saturday in a year is devoted for the students to
reflect on their service experiences from a faith-based perspective. Juniors and
seniors have their recollections as part of their theology classes. For those in third
year, they have a course on love and commitment, and their recollection focuses
on the same theme. For those in fourth year, they have a course on Catholic social
 INTERNATIONAL JOURNAL OF CHILDREN’S SPIRITUALITY   5

teaching, where students have a three-day visit in a fishing, farming, prison or

urban poor community to immerse in their work and lives. This last recollection
focuses on integrating the students’ experiences from their immersion areas.
These half-day recollections of around 50–60 persons happen on Saturdays,
either in the morning or the afternoon. It usually starts with the theme or grace
desired for the recollection, a scripture passage and explanatory points to guide
the recollection. After this 40-min talk by the recollection giver, a guest sharer
relates the religious theme to practical choices in life. After this, students are given
around half an hour for silent individual prayer and are given guide questions on
pieces of paper. After the time for prayer, they are divided into smaller groups of
five or six, where they share their immersion experiences or the fruits of their prayer
and reflection. There is a short break for the students to eat snacks.
After the break, students again come together for a guided examination of
conscience. Those who hope to avail the sacrament of reconciliation are invited
to approach the available confessors. When done, the group is divided between
those who will remain for the Holy Eucharist and those who will join the multifaith
group, mostly composed of evangelical Christians, people of other faith traditions,
agnostics and atheists. While the Catholics have the celebration of the Eucharist
inside the chapel, the multifaith group finds another suitable venue for their shar-
ing, which is guided by a facilitator invited by the OCM. Themes revolve around
the individuals’ faith journey, their hopes, dreams and living of life.
Whether part of the multifaith group or the Catholic service, all students will
be given a sheet of paper at the end that tries to review their recollection experi-
ence. At the top of the page, students evaluate in a six-point or five-point Likert
scale their participation in the recollection, the recollection’s delivery and their
attitudes as they leave the recollection. At the bottom, students reflect and write
about the dispositions they had, how they feel as they leave the recollection, and
what their image of God was.
The OCM analyses the students’ answers to the recollection review to evaluate
the programme’s delivery and impact. In the past, delivery was usually evaluated
by looking at the average grades for items such as ‘The recollection facilitator
effectively conveyed the core message of the recollection’. If there are items that
have an average below 4.5 points in the 6-point scale, these are investigated to
see how these might be improved. The data are from the first semester of the year
2016–2017, with 3,512 valid observations for all four year levels.

Outcome variable

To determine an intervention’s impact, it is important to clarify what behaviours

and attitudes the programme intervention targets. For spiritual recollections, the
desired outcome or impact is the spiritual growth and maturity of the person.
But how do people quantify or even exhibit something as intangible as spiritual
growth? This was among the core questions the Marian University’s OCM had to
6   J. E. TRINIDAD

answer. After conversations and discussions, the Office determined that the grace
or theme of the recollection is an important outcome per year level. Thus, these
statements were used as the outcome variables for each year level:

• Freshman recollection: The recollection made me appreciate God’s gifts.

• Sophomore recollection: The recollection increased my desire to do acts of
love.
• Junior recollection: The recollection moved me to deepen my relationship
with God.
• Senior recollection: The recollection made me desire to bring God to my
decision-making.

The year level’s statement approximates the goal of the recollection, and the
prayer points given were directed towards emphasising these year-level goals. In
the review form, students rated these statements on a six-point Likert scale from
strongly disagree (1) to strongly agree (6). Rather than use the mean, the study uses
the percentage of people who agreed and strongly agreed with the statements.
Thus, a dichotomous variable is created with individuals who have rated 5 or 6
being recoded as one, and all the rest who have rated from 1 to 4 being recoded
as zero. This recoding gives a more intuitive sense of the percentage of people
who were affected by the recollection.
Although this only approximates the impact of the recollection, it nonetheless
gauges the effect of the recollection on the attitudes of the students. The OCM does
not have pre-recollection and post-recollection data, so asserting that change hap-
pened because of the recollection is an untenable claim. Given the limitations, the
outcome variable still provides a sense of how many students are influenced by the
recollections, and gives a picture of the short-term impact the recollections have.

Predictor variables

Aside from learning about the percentage of people who gained from the recol-
lection, the OCM wanted to see what factors could have influenced these people.
Who are more likely to say that the recollection has had an impact on them? Three
statements in a five-point Likert scale provide possible indicators: (i) I was able to
share my reflections to the small group, (ii) I listened to the prayer points and (iii) I
was able to spend time in prayer. These three variables were determined because
of their correlations with the outcome variable.
Different from the outcome variables that are on a six-point Likert scale, the
statements for the predictor variables are on a five-point Likert scale, from Not at
all to Very much. When encoded, they took values of 1 – Not at all, 2 – A little, 3 –
Fairly, 4 – Quite and 5 – Very much, as answers to the statements like ‘I was able to
share my reflections to the small group’. Similar to the outcome variables, these are
then recoded as dichotomous variables that have a value of 1 if the answer is 4 or
5, and a value of 0 if the first three. By doing this, one is able to see the percentage
of people who were able to share, listen and pray during the recollection.
 INTERNATIONAL JOURNAL OF CHILDREN’S SPIRITUALITY   7

Other variables could play roles that affect the impact of the recollection, so
these dichotomous variables are added. They are coded as 1 if the student forms
part of this group and 0 if not. The first two variables are going to confessions and
joining the multifaith group. The variables were used for all four year levels. The
third is a dichotomous variable for being part of the morning group, since it is
possible that the time of the day had some effect on the recollection’s participants.
This was only used for the initial regression, not the subsequent ones.

Data analysis and results

Four steps were taken to analyse the data. First, descriptive statistics were per-
formed to show the percentage of recollection participants who have rated the
outcome or predictor variables highly, or have joined confession or multifaith
group. Second, correlations were done on a specific year level to see what factors
should be included for the regressions. In this case, the senior year data-set was
used. Third, logistic regressions were done with the data, and six models were
created (each one adding a predictor to the previous). Finally, the model with the
best fit for the senior year data was applied to the three other year levels. In this
data analysis, the initial data files were in Microsoft Excel format. STATA/MP 14.1
was used to recode the data, and perform descriptive statistics, correlations and
logistic regressions.

Descriptive statistics

Table 1 shows descriptive statistics for each of the four year levels in Marian
University. The top part of the table shows the percentage of students who have
gained from the recollection, based on the outcome variables determined for each
year level. There were 89% from first year students who found that the recollection
helped them appreciate God’s gifts. For sophomores, 74% found the recollection
increased their desire to do acts of love, and for juniors, 64% found the recollection
moved them to deepen their relationship with God. For seniors, the percentage
of students who desired to bring God in their decision-making was 67%. These
numbers show that a substantial majority felt that the recollection had some dis-
cernible impact for them.
The middle portion shows the percentage of people who were able to share
in small groups, listen to the recollection points and spend time in prayer. For
all four year levels, sharing was consistently rated highly since more than 80%
of students felt that they were able to share their reflections. Listening to the
recollection points was more varied between year levels as 94% of sophomores
say they listened, while only 78% of seniors affirm this. However, consistently low
for all four levels was the variable on students’ praying during time for individual
reflection. Its highest is 86% for sophomores, while its lowest is just 70% for juniors.
For the other variables, less than half of the students go to confessions during
recollections with the lowest percentage being with seniors at 29%. Participation in
8   J. E. TRINIDAD

Table 1. Descriptive statistics for recollection participants (First Semester, SY 2016–2017).

  1st Year 2nd Year 3rd Year 4th Year
Outcome variable:        
 Appreciated God’s gifts 89.18      
  Increased desire to do acts of love   74.18    
  Moved to deepen relationship with God     63.62  
 Desired to bring God to decision       67.19
Predictor variables        
 Shared during sharing time 82.64 94.53 85.62 92.16
 Listened to points 86.44 93.86 81.50 78.29
 Prayed during prayer time 75.74 86.18 69.50 75.63
  Went to confessions NA 41.17 48.00 28.83
  Joined multifaith sharing 16.29* 28.50 16.88 23.04
  Morning participants NA 62.38 NA 51.15
Number of Participants 841 1042 800 829
Note: The table shows the percentage of students who responded positively or form part of the groups mentioned.
First-year recollections do not have confessions. The morning or afternoon variable is unavailable for freshman
and junior participants. Missing variables have been dropped, owing to non-response on the outcome variable.
The outcome variable was coded 1if the student answered 5 or 6 in the Likert scale and 0 if otherwise. For the
predictors, it was coded 1 if 4 or 5 and 0 if otherwise.
Source: Marian University Office of Campus Ministry (2016–2017).
*
There was no multifaith group, but these were the people who signified they were not Catholics.

the multifaith group was also varied with 28% for sophomores and 17% for juniors.
These descriptive statistics give a sense of the profile of students’ engagement and
it also provides data on the reach of the recollections.

Correlations

Rather than have separate correlations table for all four year levels, it is more effi-
cient to focus on one year level and see the correlations between the outcome and
predictor variables. In this specific case, the senior year recollection data were used.
Table 2 presents correlations between variables. Strong correlations were seen
between praying and the outcome variable of bringing God into decisions (0.45),
and praying and listening to the recollection points (0.42). The outcome variable
is also strongly positively correlated with listening to the prayer points (0.31), and
strongly negatively correlated with joining the multifaith group (−0.29). Among
the other negative correlations, joining the multifaith group is also negatively
correlated with praying (−0.29) and going to confession (−0.35).
There is a weak correlation, however, between forming part of the morning
participants and the other variables. This means that there may not have been
any significant differences in the variation between morning and afternoon par-
ticipants. Going to confession also has weak correlations with other variables,
although substantive for praying (0.25) and bringing God to decisions (0.23). These
correlations help the OCM determine what variables to include in the regression
models, and they should not be understood as factors causing changes in the
outcome variables.
 INTERNATIONAL JOURNAL OF CHILDREN’S SPIRITUALITY   9

Table 2. Correlations table, n = 829 students.

  1 2 3 4 5 6 7
(1) Desire to bring God to decision 1.00            
(2) Shared during sharing time 0.15 1.00          
(3) Listened to points 0.31 0.32 1.00        
(4) Prayed during prayer time 0.45 0.29 0.42 1.00      
(5) Went to confessions 0.23 0.04 0.12 0.25 1.00    
(6) Joined multifaith sharing –0.29 –0.01 –0.09 –0.29 –0.35 1.00  
(7) Morning participants 0.04 0.02 0.02 0.05 0.09 –0.00 1.00
Notes: The table presents the correlations between the different variables. One can notice the strong negative cor-
relation between joining the multifaith group and most of the other variables, and the weak correlation between
being a morning participant and all the other variables.
Source: Marian University Office of Campus Ministry (2016–2017).

Logistic regression for senior year data

Since the outcome variable is a dummy that turns 1 if the person agrees or strongly
agrees to the statement that the recollection made them desire to bring God into
decisions, logistic regressions were done to see the effect of the predictors on
the odds ratio of ‘benefiting from the recollection’. Similarly, because all the other
variables were dichotomous variables, the interpretation is that belonging to this
type of group (e.g., those who were able to share openly during the recollection)
increases or decreases the odds of benefiting from the recollection.
Each model adds predictors and shows the increase in the Chi2 with the addition
of predictors. Model 1 only includes the effect of sharing, Model 2 adds the effect
of listening to points, while Model 3 adds the effect of praying. Models 4 and 5 add
the effects of going to confession and joining the multifaith group, respectively. To
ascertain if there is any variability between morning and afternoon participants,
Model 6 adds the effect of being in the morning group. All the effects were trans-
formed to odds ratio – that is, the odds of something happening over the odds
of it not happening. Thus, if the odds ratio is above 1.00 it means that belonging
to this group increases the odds of benefiting from the recollection. However, if
the odds ratio is below 1.00 it means that belonging to this group reduces these
odds.
Table 3 Model 1 shows that those who describe themselves as sharing during
the recollection are three times more likely to benefit from the recollection when
compared to those who don’t (OR 3.19, p < 0.001). This effect is made insignificant,
however, with the addition in Model 2 of the variable on listening, which signifi-
cantly predicts the gain from recollection (OR 4.14, p < 0.001). This effect is reduced
though not made insignificant with the addition of one more variable in Model 3,
that of praying during the prayer period. Those who say that they prayed during
the recollection are six times more likely to say that they desire to bring God into
prayer (OR 6.11, p < 0.001). Thus, among the three primary predictor variables,
prayer ‘participation’ strongly predicts likelihood of gaining from the recollection.
The fourth model that controls for those going to confession reduces the effect
of prayer (OR 5.13, p < 0.001) but it still retains the characteristic of the item being
10   J. E. TRINIDAD

Table 3. Logistic regression of likelihood of bringing god to decision-making, n = 829 students.

  Model 1 Model 2 Model 3 Model 4 Model 5 Model 6
Shared during sharing time 3.19*** 1.67 0.99 1.07 1.18 1.18
(0.84) (0.48) (0.31) (0.34) (0.38) (0.38)
Listened to points   4.14*** 2.43*** 2.39*** 2.49*** 2.49***
(0.77) (0.50) (0.50) (0.53) (0.53)
Prayed during prayer time     6.11*** 5.13*** 4.36*** 4.34***
(1.19) (1.02) (0.86) (0.89)
Went to confessions       2.31*** 1.75 1.73
(0.49) (0.38) (0.39)
Joined multifaith sharing         0.41*** 0.41***
(0.08) (0.08)
Morning participants           1.09
(0.19)
Intercept 0.71 0.44*** 0.29*** 0.25*** 0.34*** 0.33***
(0.18) (0.12) (0.09) (0.08) (0.10) (0.10)
LR Chi2 19.60 79.34 169.30 186.17 205.33 205.57
Notes: The table presents the odds ratio of the likelihood of bringing God to decision-making, given the predictors
in each model. For example, in model 1, those who were able to share during the sharing time are three times
more likely to bring God to decision-making. Standard errors are written in parentheses.
Source: Marian University Office of Campus Ministry (2016–2017).
*
p < 0.05; **p < 0.01; ***p < 0.001.

most predictive of benefit from the recollection. Adding a dummy for participation
in the multifaith group, Model 5 eliminates the effect of going to confessions and
sharing, and shows that those who form part of the multifaith group are less likely
to say that the recollection made them desire to bring God in their decision-mak-
ing (OR 0.41, p < 0.001). Adding the time variable, Model 6 does not substantially
change the effects from Model 5. Looking at the Chi2, the preferred model is Model
5 because the addition of the time element did not substantially increase the
variability explained by the model.

Logistic regression for four years

Since the preferred model is the fifth one, this is used to perform the logistic
regression on all four year levels. Thus, each year level’s outcome variable (e.g., for
freshmen, recollection making them appreciate God’s gifts) is regressed on the five
predictors. All four year-level regressions will have five predictors except for the
freshman recollections that did not have the confession variable.
For the freshmen recollection data on Table 4, participation through sharing,
listening and praying all increased the odds of benefitting from the recollection
(respectively: OR 2.39, 2.95 and 3.10, p < 0.001). Dissimilar to the fourth year data,
there was no single variable among the three that was the primary predictor driv-
ing the increased odds of responding positively to the outcome variable. Similar
to the senior year data, those who say that they are non-Catholics are less likely
to respond positively to the recollection’s goal (OR 0.54, p < 0.001).
For the sophomore recollections, it is interesting that the data do not fit or fol-
low the freshman or senior year’s trends. The three indicators of participation in
the recollection were not significant predictors and the only significant predictor
 INTERNATIONAL JOURNAL OF CHILDREN’S SPIRITUALITY   11

Table 4. Logistic regression on outcome variable (First Semester, SY 2016–2017).

  1st Year 2nd Year 3rd Year 4th Year
Shared during sharing time 2.39** 1.39 1.97 1.18
(0.63) (0.48) (0.53) (0.38)
Listened to points 2.95*** 0.81 2.52*** 2.49***
(0.83) (0.33) (0.62) (0.53)
Prayed during prayer time 3.10*** 1.06 5.06*** 4.36***
(0.84) (0.24) (1.01) (0.86)
Went to confessions   0.93 0.58 1.75
(0.14) (0.12) (0.38)
Joined multifaith sharing/Being non-Catholic for 1st yr. 0.54* 2.69*** 0.19*** 0.41***
(0.14) (0.60) (0.05) (0.08)
Intercept 1.09 1.97 0.29*** 0.34***
(0.28) (1.02) (0.09) (0.10)
LR Chi2 97.17 35.44 263.36 205.33
Number of participants 841 1042 800 829
Notes: The table presents the odds ratio of the likelihood of responding positively to the outcome variables. For
example, in first-year students, those who were able to pray during the prayer time are three times more likely to
have ‘benefited’ from the recollection. Standard errors are written in parentheses.
Source: Marian University Office of Campus Ministry (2016–2017).
*
p < 0.05; **p < 0.01; ***p < 0.001.

was participation in the multifaith group that actually increased the odds of one
responding positively to the outcome variable (OR 2.69, p < 0.001). Rather than
infer that this goes against the data, this peculiar result could have actually resulted
from the high participation rate in the sophomores recollection (refer to Table 1
to see that the participation rates were above 85%); this made these variables less
discriminating to show who have benefited. The relatively high participation in the
multifaith group (28.50%) could have also affected the increased odds of respond-
ing positively to the outcome variable. Additionally, the Chi2 of this model is only
35.44, which makes it so much smaller than the other’s (e.g., 263.36 for juniors).
The junior recollections’ data most closely approximate the senior year data.
The most significant predictor is prayer, where those who say that they were able
to pray during the recollection are five times more likely to respond positively
to the outcome variable (OR 5.06, p < 0.001). This is followed by listening to the
recollection points (OR 2.52, p < 0.001). Similar to seniors, participation in the mul-
tifaith group reduced the odds of responding positively to the outcome variable
of desiring a deeper relationship with God (OR 0.19, p < 0.001).

Discussion and conclusions

As part of Marian University’s formation programme for its students, the spiritual
recollections are evaluated to see if they are reaching their intended goals. First,
there has to be evidence that majority of the students are benefitting from the
recollections, codified in students agreeing that they achieved the recollection
goals. Second, there is a need to know what factors are affecting higher likelihood
of achieving these ‘goals’. Third, there is a need to understand the factors that
influence reduced likelihood of benefiting from the recollections and what could
12   J. E. TRINIDAD

be done with these. These three will be discussed from the Marian University’s
data and this discussion can be applied with recollection programmes by other
organisations.

Evaluating recollection impact

To know how many students benefit from the recollections, there were three
steps that were taken: (i) determine recollection goal, (ii) transform Likert scale to
dichotomous variables and (iii) compute the percentage of people who agreed or
strongly agreed. The most important aspect of spiritual recollections’ evaluation
is to be clear about the goal of each recollection. When the OCM met for evalu-
ation, the emphasis was to clarify the goals for each year-level recollection and
how all the elements of the recollection move towards those goals. For example,
the senior year’s recollection goal is for students to desire to bring God in their
decision-making, and because of this, the slide deck, prayer points and guest shar-
ing all contribute to this focus of bringing God to students’ decision-making. In
addition, clear recollection goals give a sense that the change brought about was
intentional and deliberate.
To quantify the impact, students encircle from a continuum of strongly disagree
to strongly agree. Rather than get the mean score that may be affected by outliers,
it is more intuitive to compute for the percentage of students who have agreed
or strongly agreed with the recollection goal being met. This is done by recoding
1–4 as zero, and 5–6 as one. With this, we get the percentage of students who have
responded affirmatively to achieving the recollection goal. From the results, we
see that more than 60% in every year level have said that the recollection goals
were met.
From a more generalised perspective, one way of evaluating the success of a
programme is through an attitudinal indicator, determined prior to the recollec-
tion. What change in attitude is expected in the recollection participants? After
determining this, the group will have to set the target percentage of participants
who agree to this indicator. When the recollection evaluation is done, campus
ministers may compute the percentage of people who agreed or strongly agreed
with the statement, and this will inform the magnitude of the programme’s impact.

Evaluating factors affecting impact

After determining the percentage of people who have responded affirmatively to

the recollection goals, a second step is to look for what factors affect higher like-
lihood of benefit from the recollection. For this study, it was shown that listening
to the recollection points and spending time in prayer increased the likelihood
of benefiting from the recollections. On the one hand, this might be understood
as people gaining from the recollection because of the spiritual power of prayer
and disposing oneself to prayer. On the other hand, this can also be understood
 INTERNATIONAL JOURNAL OF CHILDREN’S SPIRITUALITY   13

as people benefiting because they have engaged and participated – and felt that
they have participated significantly. Thus, the metacognitive understanding of
one’s participation helps them more likely to affirm that the recollections had an
effect with them (Melnick et al. 2001). These two explanations span both religious
and practical: the first focuses on the significance of prayer, while the second
highlights the importance of metacognitive understanding of one’s participation
and the effect of that participation.
Related to this is the third point about what factors could negatively influ-
ence the outcome of people benefiting from the recollection. From the logistic
regressions, it can be seen that participation in the multifaith group (or being a
non-Catholic, in the case of freshmen year) lowers the likelihood of agreeing to the
outcome variable. One possible explanation for this is that the multifaith group
is heavily composed of atheists and agnostics who will be less likely to respond
positively to an outcome variable that explicitly talks about their relationship with
God. Another explanation is that those who belong to the multifaith category
are not as effectively reached in a recollection that is still predominantly Roman
Catholic in nature. Thus, there is a need to study this further to see the pattern of
those participating in the multifaith group.
When other campus ministry offices would like to know what factors could
affect the recollection, they have to determine explanatory variables that can be
measured. Among variables that can be studied are students’ religious orienta-
tions, perceived participation in the activities (similar to what was measured here),
prior engagement (e.g., did student go to a pre-recollection training or not?), or
even the student’s course (e.g., business or engineering). When determining these
explanatory variables, the evaluator must infer how the variables help increase or
decrease the likelihood of benefiting from the recollection.
Initially, the discussion focused on the explanations for why listening and pray-
ing help increase this likelihood, and why participation in the multifaith group
decreases this likelihood. It is inferred that both participation and metacognition
with participation help in increasing the odds, while those who feel disconnected
from the liturgies and activities are less likely to participate and thus less likely to
benefit from the recollection. When other people would like to study their own
predictors, it is important for them to see the intuition behind the numbers, and
consequently, what can be done with those statistical results.

Action from evaluations

When performing program evaluations, it is one thing to let the organisers know
about the success of a programme (in this case, a spiritual recollection) and it is
another thing to actually work on improving this programme. Marian University’s
OCM actually instituted changes in its recollections after the evaluations. First,
there was a more conscious effort to highlight and continually go back to the
recollection theme or goals. Since the recollection theme for seniors is for them
14   J. E. TRINIDAD

to bring God into their decision-making, much of the prayer points are directed
towards this goal. Even the sharing of the guest sharer and the priest’s homily
are aligned with this goal. In a sense, the repetition helps with the recall and the
unconscious inculcation of the value being fostered in the recollection. To add to
this, being clear about this recollection goal also helps students clarify the purpose
of their coming to the recollection.
Second, facilitators have thought more deliberately about how to encourage
participation among different students. For example, everyone is encouraged to
write their prayers in prayer sheets, so that they may learn what it is they aspire
to have. Having prayer sheets also facilitate students who may be uncomfortable
with the 30-minute silence that is usually allotted for the individual prayer activ-
ity. Another example of encouraging participation is with the change in how the
multifaith group is facilitated. If from before it was usually facilitated as a freeform
sharing among people of different faiths, the facilitators now are more deliberate
in the present structure. There are three rounds of questions, where students get
to share to the group composed of 8–10 members. Thus, rather than have people
share about random things, the sharing is still directed towards the recollection
theme and fosters a community that tries to understand individual faith journeys.
Third, rather than think of the evaluation form as an additional activity that
students need to do, the evaluation form was refashioned as a recollection review,
where students reflect meditatively about what happened to them in the recol-
lection. From before, students would usually answer the evaluation in the fastest
manner. With this recollection review, there is time devoted for its answering and
students work on it as if collecting the fruits of the recollection. The primary focus
is not so much on the programme’s improvement but on the students’ opportunity
to reflect metacognitively about the change they experienced.
Marian University’s spiritual recollections have transitioned to clearer goals for
the recollections, improved disposition setting for prayer, more structured multi-
faith activities and more reflective rather than evaluative review of the recollection
experience. These changes were instituted because of the data presented, and
different evaluation data will show different ways for recollections to improve.
By understanding which factors predict higher or lower likelihood of benefit or
participation, facilitators could also gauge which parts to strengthen or which
parts to change or address.

Conclusion
The rise in program evaluation has been ushered by the need for greater accounta-
bility. In the field of spiritual recollections, evaluation is done to constantly improve
the programme. However, the difficulty with evaluating spiritual recollections is the
intangible nature of spiritual growth. In this attempt at modelling spiritual growth,
the focus was on the identification of outcome and predictor variables, recoding
these variables to provide a more intuitive understanding of the percentage of
 INTERNATIONAL JOURNAL OF CHILDREN’S SPIRITUALITY   15

people who gained and participated in the recollections. By showing which factors
are more likely to help students gain from the recollection, the model also provides
how changes can be instituted through the insights from the evaluation data.
If this is applied to other contexts, it is best that the organisers have a sense of
their desired goal for the recollection participants and what they infer to be pre-
dicting this desired goal. There are three things that can be done systematically.
First, organisers can recode the Likert scale to a dichotomous scale, and from this,
they could get the percentage of people who benefitted or participated in the
recollection. Second, they may create a logistic regression with the different pre-
dictor variables to see which factors are driving higher or lower likelihood of gain
from the recollection. Lastly, these results may be analysed by different members
of the team to understand how changes can be instituted from the results.
Although there are difficulties in determining predictors and identifying out-
comes, the learning process that comes along with it is an essential component.
Many things may be affecting the spiritual growth of participants – things that can-
not be measured like God’s grace and internal stirrings. This model does not deny
the importance of these unmeasured components, but this model at evaluation
approximates how recollections could actually be tilling the spiritual soil for grace
to plant its seed. In modelling evaluations for spiritual recollection, organisers can
better till this spiritual soil and let the seeds of grace grow and bear fruit.

Disclosure statement
No potential conflict of interest was reported by the author.

Notes on contributor
Jose Eos Trinidad is an instructor of Interdisciplinary Studies and Coordinator for Research and
Outreach of the Ateneo de Manila Institute for the Science and Art of Learning of Teaching
(Ateneo SALT Institute). He received his bachelor’s degree from the same university and his
graduate degree at the University of Chicago. He is interested in noncognitive factors that
affect learning and how these factors affect education systems.

ORCID
Jose Eos Trinidad   http://orcid.org/0000-0001-9986-8683

References
Angrist, Joshua D.,  and Alan B. Krueger. 1999. “Chapter 23 – Empirical Strategies in Labor
Economics.” In Handbook of Labor Economics, edited by Orley C. Ashenfelter  and David Card, 3
vols. 1277–1366. Amsterdam: North Holland Publisher.
Angrist, Joshua D., Parag A. Pathak, and Christopher R. Walters. 2013. “Explaining Charter School
Effectiveness.” American Economic Journal: Applied Economics 5 (4): 1–27.
16   J. E. TRINIDAD

Astin, Alexander W., Helen S. Astin, and Jennifer A. Lindholm. 2011. Cultivating the Spirit: How
College Can Enhance Students’ Inner Lives. San Francisco, CA: Jossey-Bass.
Batton, Jennifer. 2003. “Cost-Benefit Analysis of CRE Programs in Ohio.” Conflict Resolution
Quarterly 21 (1): 131–133.
Beare, Paul, James Marshall, Colleen Torgerson, Susan Tracz, and Robin Chiero. 2012. “Toward
a Culture of Evidence: Factors Affecting Survey Assessment of Teacher Preparation.” Teacher
Education Quarterly 39 (1): 159–173.
Brock, William A., Steven N. Durlauf, and Kenneth D. West. 2007. “Model Uncertainty and
Policy Evaluation: Some Theory and Empirics.” Journal of Econometrics, the Interface between
Econometrics and Economic Theory 136 (2): 629–664. doi:10.1016/j.jeconom.2005.11.009.
Castiglia, Beth, and David Turi. 2011. “The Impact of Voluntary Accountability on the Design of
Higher Education Assessment.” Academy of Educational Leadership Journal 15 (3): 119–130.
Coe, Robert, James Arthur, and Larry V. Hedges. 2017. Research Methods and Methodologies in
Education. Thousand Oaks, CA: Sage.
Dobbie, Will, and Roland G. Fryer Jr. 2011. “Are High-Quality Schools Enough to Increase
Achievement among the Poor? Evidence from the Harlem Children’s Zone.” American Economic
Journal: Applied Economics 3 (3): 158–187.
Durlak, Joseph A., Roger P. Weissberg, Allison B. Dymnicki, Rebecca D. Taylor, and Kriston B.
Schellinger. 2011. “The Impact of Enhancing Students’ Social and Emotional Learning: A
Meta-Analysis of School-Based Universal Interventions.” Child Development 82 (1): 405–432.
doi:10.1111/j.1467-8624.2010.01564.x.
Emshoff, James G. 2008. “Researchers, Practitioners, and Funders: Using the Framework to Get Us
on the Same Page.” American Journal of Community Psychology 41 (3–4): 393–403. doi:10.1007/
s10464-008-9168-x.
Fitz-Gibbon, Carol Taylor, and Lynn Lyons Morris. 1996. “Theory-Based Evaluation.” Evaluation
Practice 17 (2): 177–184. doi:10.1016/S0886-1633(96)90024-0.
Garces, Eliana, Duncan Thomas, and Janet Currie. 2002. “Longer-Term Effects of Head Start.”
American Economic Review 92: 999.
Germán, Miguelina, Nancy A. Gonzales, Stephen G. West, and Lorey A. Wheeler. 2017. “An
Experimental Test of the Bridges to High School Intervention on Harsh Parenting and Early
Age Intercourse among Mexican American Adolescents.” Cultural Diversity and Ethnic Minority
Psychology 23 (3): 362–372. doi:10.1037/cdp0000131.
Geven, Koen, Jan Skopek, and Moris Triventi. 2017. “How to Increase PhD Completion Rates?
An Impact Evaluation of Two Reforms in a Selective Graduate School, 1976–2012.” Research
in Higher Education, October: 1–24. doi:10.1007/s11162-017-9481-z.
Grant, Janet. 2002. “Learning Needs Assessment: Assessing the Need.” BMJ 324 (7330): 156–159.
doi:10.1136/bmj.324.7330.156.
Hamermesh, Daniel S. 2017. “Replication in Labor Economics: Evidence from Data, and What It
Suggests.” American Economic Review 107 (5): 37–40. doi:10.1257/aer.p20171121.
Hatch, Robert L., Mary Ann Burg, Debra S. Naberhaus, and Linda K. Hellmich. 1998. “The Spiritual
Involvement and Beliefs Scale: Development and Testing of a New Instrument.” The Journal
of Family Practice 46 (6): 476–486.
Heckman, James, Rodrigo Pinto, and Peter Savelyev. 2013. “Understanding the Mechanisms
through Which an Influential Early Childhood Program Boosted Adult Outcomes.” American
Economic Review 103 (6): 2052–2086. doi:10.1257/aer.103.6.2052.
Hult, Agneta, and Charlotta Edström. 2016. “Teacher Ambivalence towards School Evaluation:
Promoting and Ruining Teacher Professionalism.” Education Inquiry (Co-Action Publishing) 7
(3): 305–325. doi:10.3402/edui.v7.30200.
Ingersoll, Richard M. 2003. Who Controls Teachers’ Work: Power and Accountability in America’s
Schools. Cambridge, MA: Harvard University Press.
 INTERNATIONAL JOURNAL OF CHILDREN’S SPIRITUALITY   17

Jayaraman, Rajshri, and Dora Simroth. 2015. “The Impact of School Lunches on Primary School
Enrollment: Evidence from India’s Midday Meal Scheme.” The Scandinavian Journal of Economics
117 (4): 1176–1203. doi:10.1111/%28ISSN%291467-9442/issues.
Block Joy, Amy, George Goldman, and Vijay Pradhan. 2006. “Cost-Benefit Analysis Conducted
for Nutrition Education in California.” California Agriculture 60 (4): 185–191.
Kirkpatrick, James D., and Wendy Kayser Kirkpatrick. 2016. Kirkpatrick’s Four Levels of Training
Evaluation. Alexandria, VA: ATD Press.
Korth, Sharon J. 2001. “Consolidating Needs Assessment and Evaluation Saving Time and Money.”
Performance Improvement 40 (1): 38–43. doi:10.1002/pfi.4140400110.
Laudel, Grit. 2006. “The ‘Quality Myth’: Promoting and Hindering Conditions for Acquiring
Research Funds.” Higher Education 52 (3): 375–403. doi:10.1007/s10734-004-6414-5.
Marcella, Rita, Hayley Lockerbie, and Lyndsay Bloice. 2016. “Beyond REF 2014: The Impact of
Impact Assessment on the Future of Information Research.” Journal of Information Science
42 (3): 369–385. doi:10.1177/0165551516636291.
McDavid, James C., Irene Huse, and Laura R. L. Hawthorn. 2012. Program Evaluation and
Performance Measurement: An Introduction to Practice. Thousand Oaks, CA: Sage.
Melnick, Gerald, George De Leon, George Thomas, David Kressel, and Harry K. Wexler. 2001.
“Treatment Process in Prison Therapeutic Communities: Motivation, Participation, and
Outcome.” The American Journal of Drug and Alcohol Abuse 27 (4): 633.
Ngware, Moses Waithanji, Benta Abuya, Moses Oketch, Kassahun Admassu, Maurice Mutisya, and
Peter Musyoka. 2015. “Randomized Impact Evaluation of Education Interventions: Experiences
and Lessons from a Reading to Learn Intervention in East Africa.” International Journal of
Research & Method in Education 38 (4): 430–451. doi:10.1080/1743727X.2014.965252.
Paull, Megan, Craig Whitsed, and Antonia Girardi. 2016. “Applying the Kirkpatrick Model:
Evaluating an ‘Interaction for Learning Framework’ Curriculum Intervention.” Issues in
Educational Research 26 (3): 490–507.
Petrosino, Anthony. 2000. “Answering the ‘Why’ Question in Evaluation: The Causal-Model
Approach.” Canadian Journal of Program Evaluation 15 (1): 1–24.
Pinfold, Vanessa, Hilary Toulmin, Graham Thornicroft, Peter Huxley, Paul Farmer, and Tanya
Graham. 2003. “Reducing Psychiatric Stigma and Discrimination: Evaluation of Educational
Interventions in UK Secondary Schools.” British Journal of Psychiatry 182 (04): 342–346.
doi:10.1192/bjp.182.4.342.
Ravitch, Diane. 2010. The Death and Life of the Great American School System. New York, NY:
Basic Books.
Reio Jr, Thomas G., Tonette S. Rocco, Douglas H. Smith, and Elegance Chang. 2017. “A Critique
of Kirkpatrick’s Evaluation Model.” New Horizons in Adult Education & Human Resource
Development 29 (2): 35–53.
Rossi, Peter H., Mark W. Lipsey, and Howard E. Freeman. 2004. Evaluation: A Systematic Approach.
Thousand Oaks, CA: Sage.
Royse, David, Bruce A. Thyer, and Deborah K. Padgett. 2015. Program Evaluation: An Introduction
to an Evidence-Based Approach. Belmont: Thomson Learning.
Schneller, Beverly E., Martha W. Minardi, and Todd L. Lake. 2016. “Assessing Spiritual Growth and
Development at Belmont University.” Assessment Update 28 (1): 1–16. doi:10.1002/au.30043.
Stoneberg, Bert D. 2015. “Real Cost-Benefit Analysis is Needed in American Public Education.”
Practical Assessment, Research & Evaluation 20 (15): 1–4.
Tiano, Noel, and Kath Maclean. 2015. “Assessing the Value of Chapel Services at a Mental Health
Hospital: An Exploratory Study.” Health and Social Care Chaplaincy 3 (2): 191–200.
Unterhalter, Elaine. 2017. “Negative Capability? Measuring the Unmeasurable in Education.”
Comparative Education 53 (1): 1–16. doi:10.1080/03050068.2017.1254945.
18   J. E. TRINIDAD

Vanclay, Frank. 2003. “International Principles for Social Impact Assessment.” Impact Assessment
and Project Appraisal 21 (1): 5–12. doi:10.3152/147154603781766491.
Yamashita, Takashi, Erick B. López, Jennifer R. Keene, and Jennifer M. Kinney. 2015. “Predictors
of Adult Education Program Satisfaction in Urban Community-Dwelling Older Adults.”
Educational Gerontology 41 (11): 825–838. doi:10.1080/03601277.2015.1050909.