Professional Documents
Culture Documents
Background In problem-based learning (PBL), problems Results The results showed that students were able to
represent the starting point of students’ learning activit- distinguish PBL problems that were too simple and
ies. Therefore, the quality of these problems should be those that were too well-structured, but found it
high, in that they should be of an adequate level of difficult to distinguish problems that were too complex
complexity and structuredness. Previous research has or too ill-structured.
proposed several guidelines for constructing problems, Conclusion The questionnaire may be used to measure
but some of them are rather vague and are not based on the levels of complexity and structuredness of a
empirical evidence. The present study aimed to validate a problem as perceived by students and can provide
short questionnaire that can be used to assess the degree teachers with feedback about the quality of problems.
of complexity and structuredness of PBL problems.
Keywords education, medical, undergraduate ⁄
Method This paper outlines Jonassen’s theory, on *methods; problem based learning ⁄ *methods ⁄
which the questionnaire is based, and its relation- standards; reproducibility of results; questionnaires;
ship and applicability to PBL problems. The question- guidelines; Netherlands.
naire was validated by means of confirmatory factor
Medical Education 2003;37:1001–1007
analysis.
In the present study, Jonassen’s theory7 was used to complexity is: Because the problems contain too many
develop a short questionnaire to measure and assess the aspects, the coherence is unclear (item 6). Most of the
2 main characteristics of PBL problems: complexity statements had to be answered on a scale ranging from
and structuredness. The present study aimed to valid- 1 to 5 (1 ¼ strongly disagree, 5 ¼ strongly agree). An
ate this questionnaire. example of a statement on structuredness is: Problems
often contain too many solutions, which makes it
difficult to gain insight into the learning material
Method
(item 7). Six items were formulated to measure com-
plexity (items 1–6) and 6 items were formulated to
Subjects
measure structuredness (items 7–12). All items are
The study was conducted during the academic year presented in Table 1. Furthermore, students were asked
2001–02 at the Medical School at Maastricht Univer- to indicate on a 5-point scale whether complexity
sity, the Netherlands. All first year students were and structuredness were important criteria for determin-
involved in the study (n ¼ 244). The total response ing the quality of PBL problems (1 ¼ strongly disagree,
rate was 75%. 5 ¼ strongly agree). Finally, 1 open-ended question was
included in which students were asked to define the
characteristics that determine the quality of problems.
Instrument
Table 1 Number of items, number of students (n), minimum and maximum scores, mean score (scale 1–5) and standard deviation (SD)
AMOS program was used to test the fit of the data to the Table 3 Two-factor model
model.11 In addition, students’ answers on the open-
ended question were analysed. Data Chi- Cmin ⁄
set n square d.f. P d.f. RMR GFI AGFI
Results
Total 183 242.138 53 0.000 4.403 0.100 0.787 0.698
Descriptive statistics
RMR ¼ root mean square residual; GFI ¼ goodness-of-fit index;
The average scores on the items varied between 1Æ87 AGFI ¼ adjusted goodness-of-fit index.
(SD ¼ 0Æ84) and 2Æ74 (SD ¼ 0Æ87) (Table 1).
Students were asked to rate whether complexity and
1 too simple;
structuredness were important characteristics in deter-
2 too difficult, and
mining the quality of PBL problems. The mean score
3 too well-structured.
for complexity was 3Æ28 (SD 0Æ97, scale 1–5, range
1–5). The mean score for structuredness was 3Æ83 (SD Factor 1 contained items that were formulated as too
0Æ81, scale 1–5, range 2–5). The results are presented in simple (items 1–3). Factor 2 contained items that were
Table 2. formulated as too complex and too ill-structured (items
4–9). Factor 3 contained items that were formulated as
too well-structured (items 10–12). To assess this
Construct validity
3-factor model, a confirmatory factor analysis was
To assess the construct validity of the instrument, a conducted again. The results of the 3-factor model
confirmatory factor analysis was conducted. A confirm- (too simple, too difficult and too well-structured) were:
atory factor model is supposed to fit if the following chi-square [51 d.f.] ¼ 86Æ534, P ¼ 0Æ001, a root mean
conditions are met: square residual of 0Æ047, a goodness-of-fit index of
0Æ928 and an adjusted goodness-of-fit index of 0Æ890
1 the chi-square divided by the degrees of freedom
(Table 4). Thus, this 3-factor model met all the
should be lower than 2 and the P-value should differ
conditions of Saris and Stronkhorst.12 The data set
from 0;
was split up in order to further cross-validate the
2 the root mean square residual should be lower than
proposed model. Subset A consisted of a random set of
0Æ07, and
92 students and subset B consisted of the remaining 91
3 the goodness-of-fit index and the adjusted goodness-
students. When these 2 subsets were analysed, they
of-fit-index should be higher than 0Æ80.12
were seen to meet all the conditions of Saris and
A 2-factor model containing the factors complexity Stronkhorst12 (Table 4). The results of the confirmat-
(items 1–6) and structuredness (items 7–12) was tested. ory factor analysis indicated that the 3-factor model
The results of the 2-factor model were: chi-square (53 showed a good fit, because it satisfied all the conditions
d.f.) ¼ 242Æ138, P ¼ 0Æ000, a root mean square resid- specified by Saris and Stronkhorst.12
ual of 0Æ100, a goodness-of-fit index of 0Æ787 and an Table 5 presents the descriptive statistics of the
adjusted goodness-of-fit index of 0Æ698 (Table 3). 3-factor model. The mean scores were 2Æ3 (SD 0Æ61)
Thus, this 2-factor model met none of the conditions for too simple, 2Æ5 (SD 0Æ53) for too difficult and 2Æ5
described by Saris and Stronkhorst.12 (SD 0Æ61) for too well-structured. As the items were
A new model was constructed consisting of the formulated as extremes, the average scores indicated
following 3 factors: that students did not perceive the PBL problems they
Table 2 Number of respondents (n), minimum and maximum scores, mean score (scale 1–5), standard deviation (SD)
RMR ¼ root mean square residual; GFI ¼ goodness-of-fit index; AGFI ¼ adjusted
goodness-of-fit index.
Table 5 Number of items, number of students (n), minimum and maximum scores, mean score (scale 1–5), standard deviation (SD) and
the coefficient alpha per factor
were confronted with in their first year of study as too as referring to structuredness. Besides complexity and
simple or too well-structured, nor as too difficult. structuredness, 3 other desirable characteristics of PBL
Furthermore, alpha coefficients were computed. These problems were mentioned. The first was that the
were 0Æ53 for factor 1, 0Æ69 for factor 2, 0Æ47 for factor 3 problems have to be challenging by relating them to
and 0Æ60 for all factors. practice. One student said: Problems could be made
The correlations between the factors are presented in interesting by turning them into clinical cases. The
Table 6. The correlations varied between 0Æ004 and second frequently mentioned characteristic was a
0Æ183, with the highest correlation found between the problem must have a meaningful relationship to the
factors too simple and too ill-structured. other problems in the unit. One student wrote: The
Additional information was obtained by studying the problems discussed in a unit have to be connected.
statements made by students about characteristics that The third characteristic was that the problems have to
determine the quality of a problem. Of the character- fit in with students’ prior knowledge. One student
istics examined in the present study, students men- wrote: A problem must fit in with prior knowledge, i.e.
tioned complexity and structuredness, although not in a problem must not consist of only known or fully
exactly the same terms. One student wrote: A well- unknown facts.
constructed problem contains just enough learning
cues, i.e. not too many or too few. This statement
Discussion
can be regarded as referring to complexity. Another
student said: It has to be obvious which direction the The aim of this study was to validate a questionnaire
problem is leading to. This statement can be regarded measuring 2 characteristics of PBL problems: com-
plexity and structuredness, based on Jonassen’s the-
ory.7 A confirmatory factor analysis indicated that a
Table 6 Correlations between the 3 factor scores 3-factor model fitted the data better than a 2-factor
model, as the 3-factor model met all the conditions
Factor 1 Factor 2 Factor 3 specified by Saris and Stronkhorst,12 indicating a good
fit. The 3-factor model consisted of the following 3
1 Too simple ⁄ 0.004 0.183* factors: too simple, too difficult and too well-struc-
2 Too difficult ⁄ ⁄ 0.116 tured. The data showed that students were able to
3 Too well-structured ⁄ ⁄ ⁄ distinguish between too simple and too well-structured.
An example of a problem that is too simple is presented
*Correlation is significant at the 0.05 level (2-tailed). in Section 1, Appendix 1. This problem deals with
popular health food products. Although the manufac- In conclusion, the data demonstrated that this short
turers of these products have commercial interests, the questionnaire can be used to measure 2 main charac-
product information they provide to consumers has to teristics of PBL problems: whether or not a problem is
be true. In the problem, a link to the Internet site of too simple and whether or not it is too well-structured.
such a popular health product is presented. A substan- It is, furthermore, possible to measure whether a
tial proportion of the students rated this problem as too problem is too difficult in terms of it being too
simple, but the structuredness seemed to be adequate. complicated and too ill-structured. However, the
The problem was too simple because the coherence results also indicated that the students had difficulty
between the various aspects of this problem was too in distinguishing between PBL problems that were too
easy to state (item 2). Furthermore, the problem complex or too ill-structured. A possible explanation is
contained aspects that were clearly too obviously that, as mentioned in the introduction, ill-structured
related (item 3). Thus, this problem was rated as being problems are usually more complex than well-struc-
too simple but adequately structured. tured problems, which means that students will not
However, the data also showed that it was difficult encounter many simple, ill-structured problems, or very
for students to differentiate between problems that complex but well-structured problems. Hence, too
were too complex or too ill-structured, which corres- complex and too ill-structured represent 1 factor
ponds to factor two. These problems were perceived as instead of 2 separate factors. Additionally, if a problem
being too difficult. The problem presented in Section 2, was too ill-structured and too complex, the students
Appendix 1 illustrates this. This problem concerns a had difficulty in dealing with it, because it did not fit in
young woman who suffers from fatigue and slightly with the students’ level of prior knowledge.
disturbed bowel movements. The problem includes An important limitation of the present study is that
various medical terms. The problem is too complex the results are based on students’ perceptions only.
because it contains too many different elements and the Teachers could also be asked to rate the structuredness
relationship between the elements is not always clear and complexity of problems using the short question-
(items 3 and 6). Because there are too many ways of naire developed in this study. Future research could
dealing with the problem, it was not easy for the additionally focus on investigating whether students’
students to get a grip on the learning material (item 9). and teachers’ opinions about the quality of PBL
In other words, the problem was too complex and too problems differ or not.
ill-structured. It was too difficult for the students to Finally, the questionnaire as developed and validated
address this problem, probably because it did not in this study can be used to detect whether PBL
adequately fit in with the students’ level of prior problems, in students’ opinions, are too simple, too
knowledge. Finally, the correlations between the factors well-structured or too difficult. The scores on the items
in the 3-factor model were low, which implied that the in the questionnaire can provide detailed feedback
factors measured different aspects. about a new PBL problem’s complexity and structu-
Finally, our data showed that students found structu- redness that can be used to improve the problem.
redness (mean 3Æ83) more important than complexity However, it is also necessary to collect information
(mean 3Æ28) (Table 2). A possible explanation for this about the other factors that determine the quality of a
is that students preferred some degree of structured- PBL problem.
ness, because this made it easier for them to select an
efficient approach with which to address the problem.
Contributors
The information obtained from the statements students
made about characteristics that determine the quality of All authors contributed to and participated in the
problems confirmed the above results. Although the 3 writing of this paper.
additional characteristics mentioned by students were
not investigated in this study, it would seem worthwhile
Funding
to find more evidence for these characteristics in future
research. The present study represents an initial None.
attempt to develop a validated instrument and is meant
to indicate only 2 of the characteristics that determine
References
the quality of PBL problems. In future research it
would be valuable to develop a validated instrument 1 Norman GR, Schmidt HG. The psychological basis of prob-
with which to provide feedback on other characteristics lem-based learning: a review of the evidence. Acad Med
of PBL problems. 1992;67 (9):557–65.
2 Gijselaers WH, Schmidt HG. Development and evaluation of and external worlds. Therefore, according to the
a causal model of problem-based learning. In: Nooman ZH, experts, this is the area to focus on if we want to
Schmidt HG, Ezzat ES, eds. Innovation in Medical Education: bring our bodies into balance. What is this border
an Evaluation of its Present Status. New York: Springer Pub-
between the internal and external world in the bowels
lishing Co 1990.
that is influenced by health food products like Yakult?
3 van Berkel HJM, Schmidt HG. Motivation to commit oneself
And how does such an amazing product work? It is
as a determinant of achievement in problem-based learning.
Higher Educ 2000;40:231–42. highly appealing to use Yakult’s product information or
4 van den Hurk MM, Wolfhagen HAP, Dolmans DHJM, van its commercials to find out about this. The information
der Vleuten CPM. Student-generated learning issues: a guide may be coloured, but the manufacturers are not
for individual study? Educ for Health 1999;12 (2):213–21. allowed to deceive the public! (See also http://www.
5 Dolmans DHJM, Snellen-Balendong HAM, Wolfhagen HAP, yakult.nl)
van der Vleuten CPM. Seven principles of effective case
design for a problem-based curriculum. Med Teach 1997;19
(3):185–9. Section 2
6 Majoor GD, Schmidt HG, Snellen-Balendong HAM, Moust Mirjam Janssen, who is 23 years old, visits her GP
JHC, Stalenhoef-Halling B. Construction of problems for
because she has felt extremely tired lately. Mirjam has a
problem-based learning. In: Nooman ZH, Schmidt HG, Ezzat
very busy job. She also exercises a lot because she is in
ES, eds. Innovations in Medical Education: an Evaluation of its
Present Status. New York: Springer Publishing Co 1990. training for her first marathon. She has the impression
7 Jonassen DH. Toward a design theory of problem solving. that her physical condition has decreased slightly over
Educational Technol Res Dev 2000;48 (4):63–85. the last couple of months, and despite the use of
8 Funke J. Solving complex problems: exploration and control nutrition supplements and vitamins, she seems to
of complex systems. In: Sternberg RJ, Frensch PA, eds. become breathless sooner when she is training. Accord-
Complex Problem Solving. Principles and Mechanisms. Hillsdale, ing to her fellow sportswomen and sportsmen, she looks
New Jersey: Lawrence Erlbaum Associates 1991. a bit pale. Her appetite is good, but her eating pattern is
9 Gick ML, Holyoak KJ. The cognitive basis of knowledge very irregular because of her busy job. Her weight has
transfer. In: Cormier SM, Hagman JD, eds. Transfer of
dropped by 2 kg, but she does not mind that. Apart
Learning: Contemporary Research and Applications. San Diego:
from contraceptive pills, she does not use any medica-
Academic Press 1987.
tion. Her menstruation is normal. Sometimes her
10 Schmidt HG, Moust JHC. Probleem Gestuurd Onderwijs. Pra-
ktijk en Theorie. [Problem-based Learning. Practice and Theory.]
abdomen feels swollen and she is a bit windy, but she
Groningen: Wolters-Noordhoff 1998. attributes that to her irregular eating pattern. Her faeces
11 Arbuckle JL. Amos Users’ Guide. Chicago: Small Waters are often thin, but there is no blood in them. A physical
Corporation 1997. examination does not reveal anything unusual. The
12 Saris W, Stronkhorst H. Causal Modelling in Non-experimental assistant measures Hb, and finds it to be 6Æ2 mmol ⁄ L.
Research. Amsterdam: Sociometric Research Foundation 1984. Mirjam is given a prescription for iron supplement
therapy. Her faeces have to be examined and Mirjam
Received 13 December 2002; editorial comments to authors 6 March brings some to the assistant the next day. They are
2003; accepted for publication 16 May 2003 tested for the presence of blood (negative) and lipids
(positive). Mirjam asks what this positive result
Appendix 1 means. The GP discusses this by phone with a gastro-
entrologist, who proposes additional blood examination
Section 1 and an oesophago-gastro-duodenoscopy, with biopsies
to be taken from the small intestine for microscopical
If we have to believe the manufacturers of so called investigation. One week later, Mirjam visits the gast-
health food products, our diet must be quite roentrologist to discuss the results. He tells her that
unhealthy! We are strongly advised to detoxify our the results of the blood tests show that her complaints
bodies from time to time to restore the balance in them. are due to iron deficiency. In addition, antibodies
In view of the commercial context of such messages, that react to gliadine and endomysium have formed.
some reservations seem justified. What is wrong In the duodendal biopsies, the pathologist observes
with our unhealthy food? According to the product lymphatic infiltrate and atrophy of the mucous
information, an important level of disturbance in the membrane.
bowels takes place at the border between the internal