You are on page 1of 31

National Medical Admission Test (NMAT)

Center for Educational Measurement, Inc.

The test of excellence

NMAT
Objective: to upgrade the selection of applicants to medical schools Description: an instrument that measures several abilities deemed necessary for success in medical training and practice

Center for Educational Measurement, Inc.

The test of excellence

NMAT
Abilities measured: verbal comprehension inductive reasoning quantitative ability perceptual acuity knowledge and analytical proficiency in the sciences
Center for Educational Measurement, Inc.

The test of excellence

Test Scores
8 subtest scores APT Composite SA Composite General Performance Score (GPS)

Center for Educational Measurement, Inc.

The test of excellence

Test Development Concerns


1. Relevance of the areas measured by the test to the curricula of premedical courses and the medical course itself 2. Comparability of test scores from different test forms across the years

Center for Educational Measurement, Inc.

The test of excellence

Test Development Procedure


1. 2. 3. 4. 5. 6. Content specification Item writing and review Pretesting and item analysis Test assembly Test administration Score scales generation & equating
6

Center for Educational Measurement, Inc.

The test of excellence

Bases for a Two-Part Test


1. PACT key aptitudes for the medical course 2. Academic requirements of medical schools 3. Possibility of test bias

Center for Educational Measurement, Inc.

The test of excellence

Test Form Equating


The process of making the score scales of different test forms equivalent to be able to directly compare the scores from the different test forms

Center for Educational Measurement, Inc.

The test of excellence

Test Form Equating


NMAT subtest standard scores are comparable across subtests and across test forms APT and SA composite scores are comparable with each other and across test forms GPSs are comparable across test forms
Center for Educational Measurement, Inc.

The test of excellence

Test Research Concurrent Validity Study


Multiple Correlations APT subtest SA subtest scores & GPA scores & GPA College freshmen .61 .54 College seniors .36 .33 Sample

Center for Educational Measurement, Inc.

The test of excellence

10

Test Research NMAT Highlights 19861997


NMAT: Overall performance remained relatively stable across time. PLE: Overall performance declined across the years. NMAT & PLE relationship: Students who were selected on the basis of the NMAT had a better chance of passing the PLE.
Center for Educational Measurement, Inc.

The test of excellence

11

Test Research NMAT Scores of 2006 PLE Passers


PLE passers w/ NMAT taken once (N = 1,176): GPS Mean = 584; SD = 87
GPS Distribution on Quality Indices GPS Range QI Range % 524800 476523 HAE A 76 14

200475
Center for Educational Measurement, Inc.

VPLA

10
12

The test of excellence

Test Research NMAT Scores of 2006 PLE Passers


PLE passers w/ NMAT retaken (N = 182): GPS on retake not significantly different from initial GPS

Center for Educational Measurement, Inc.

The test of excellence

13

Future Direction
Development of shorter test forms that produce better ability estimates and efficiently predict academic performance in the medical course

Center for Educational Measurement, Inc.

The test of excellence

14

Predictive Validity Study


The minimum requirement for NMAT to be useful in decision-making is an empirical demonstration of its predictive validity. Predictive validity refers to the efficiency of NMAT in forecasting any measure of academic performance in medical school.
Center for Educational Measurement, Inc.

The test of excellence

15

Technically, predictive validity of NMAT is expressed as a bivariate or multivariate correlation coefficient of NMAT scores and college grades.
The higher this correlation coefficient, the more confidence can be placed on the NMAT as a valid predictor of academic performance.

Center for Educational Measurement, Inc.

The test of excellence

16

The variables
Predictor variable
1. NMAT General Performance Score (NGPS)

Dependent or Criterion variables


2. First year college grade point average (FGPA)
based on first year academic subjects

3. Overall college grade point average (OGPA)


based on ALL academic subjects
Center for Educational Measurement, Inc.

The test of excellence

17

The validation samples


Medical school applicants who were accepted from SY1997-98 to SY2002-03 Three (3) medical schools A, B, and C; located in Luzon, Visayas, and Mindanao; respectively, participated in this validity study The final sample sizes include only students with complete data on all the three variables under study: School A (Luzon) 744

School B (Visayas)
School C (Mindanao)
Center for Educational Measurement, Inc.

205
232
18

The test of excellence

Data Analysis
Computation of weighted GPAs using the prescribed number of hours per academic subject as weights Computation of correlation coefficients between NMAT test scores and a criterion of medical school performance The accuracy of prediction is usually represented by the correlation coefficients. That is, the extent to which scores on the NMAT are correlated to success in medical college is the extent to which the test fulfills its predictive function.
Center for Educational Measurement, Inc.

The test of excellence

19

Adjustment of computed correlation coefficients due to restriction of NGPS within the range above cut-off scores used by schools (which tends to severely decrease the correlation of NGPS with other variables.) Adjusted or otherwise, the values of correlation coefficients are always interpreted within the context in which it was computed. Coefficients as low as .30 are generally considered appropriate and valid for predictive purposes (Bobko, 1995).
Center for Educational Measurement, Inc.

The test of excellence

20

Computation of the coefficient of determination, d, which indicates the proportion of variance in the FGPA or in the OGPA that is predicted by NGPS Derivation of prediction equation which estimates the FGPA or OGPA for a given NGPS Generating expectancy table which estimates the probability of a medical school applicant to attain certain FGPA or OGPA based on his/her NGPS
Center for Educational Measurement, Inc.

The test of excellence

21

Results
TABLE 2. Pearson correlation coefficients School A Variable pair Restricted Adjusted NGPS x FGPA NGPS x OGPA r12 r13 -0.375 -0.373 -0.545 -0.542 School B
Restricted Adjusted

School C
Restricted Adjusted

-0.347 -0.346

-0.560 -0.558

-0.380 -0.296

-0.604 -0.496

Performance in NMAT is moderately correlated with academic performance in the first year of medical education as well as with performance over three or four years of academic subjects.

Center for Educational Measurement, Inc.

The test of excellence

22

TABLE 3. Coefficients of determination School A School B Variable pair NGPS x FGPA NGPS x OGPA d12 d13
0.2966 0.2940 0.3131 0.3112

School C
0.3648 0.2461

Table 3 shows that around 30% of the total variation in academic performance can be explained by performance in the NMAT. In other words, the use of the NMAT GPS as predictor makes forecasting 30% to 36% better than mere random guessing.

Center for Educational Measurement, Inc.

The test of excellence

23

School A:

School B:
School C:

Predicted FGPA Predicted OGPA Predicted FGPA Predicted OGPA Predicted FGPA Predicted OGPA

= = = = = =

-0.0025 (NGPS) + 3.8076 -0.0020 (NGPS) + 3.5354 -0.0026 (NGPS) + 3.8048 -0.0021 (NGPS) + 3.6110 -0.0023 (NGPS) + 3.7591 -0.0011 (NGPS) + 2.9049

The prediction equations can be used to estimate the first year GPA and the overall academic performance of a medical student applying for admission to medical School A, B, or C.
Center for Educational Measurement, Inc.

The test of excellence

24

TABLE 5. Index of predictive efficiency School B Predicted variable School A FGPA OGPA 16.13% 15.98% 17.12% 17.00%

School C 20.30% 13.17%

Table 5 shows that the prediction of academic performance in the three medical schools based on the derived prediction equations is about 13% to 20% better than prediction based on knowledge of the mean of the obtained FGPA or OGPA.
Center for Educational Measurement, Inc.

The test of excellence

25

TABLE 7.2. Expectancy or probability of obtaining a particular FGPA at different ranges of NGPS in School A FGPA 1.75 & 1.76 2.01 2.26 2.51 2.76 NGPS Below -2.00 -2.25 -2.50 -2.75 -3.00 Below 501 501-550 551-600 601-650 651-700 Above 700 1% 2% 1% 11% 4% 4% 10% 15% 26% 44% 10% 19% 20% 29% 40% 22% 34% 34% 35% 27% 20% 11% 46% 35% 27% 22% 9% 11% 6% 9% 7% 5% 4%

Students who obtain an NMAT GPS between 651 and 700 have a 27 percent probability of attaining a first year GPA of 2.00 or higher in School A.
Center for Educational Measurement, Inc.

The test of excellence

26

TABLE 7.3. Expectancy or probability of obtaining a particular OGPA at different ranges of NGPS in School A FGPA 1.75 & 1.76 2.01 2.26 2.51 2.76 NGPS Below -2.00 -2.25 -2.50 -2.75 -3.00 Below 501 501-550 551-600 601-650 651-700 Above 700 2% 1% 5% 8% 12% 35% 4% 16% 17% 26% 42% 50% 36% 38% 46% 36% 35% 30% 54% 43% 28% 26% 11% 15% 4% 2% 3% 2%

1% 2% 5%

Students who obtain an NMAT GPS between 651 and 700 have also 12 percent probability of attaining an overall GPA of 2.00 or higher in School A.
Center for Educational Measurement, Inc.

The test of excellence

27

NMAT subtests that are predictive of FGPA and OGPA:


School A: FGPA OGPA PHYS, BIOL, CHEM, SSCI, VCOM CHEM, SSCI, PHYS, QUAN

School B: FGPA OGPA


School C: FGPA OGPA
Center for Educational Measurement, Inc.

BIOL, PHYS, VCOM BIOL, VCOM, PHYS


SSCI, CHEM, PHYS SSCI, CHEM

The test of excellence

28

Summary and Conclusion


The use of the NMAT GPS as predictor makes forecasting 30% to 36% better than mere random guessing Predicting future academic performance based on NMAT GPS and using the derived prediction equations is about 13% to 20% more accurate than predicting based on knowledge of the mean of the obtained FGPA or OGPA of the validation sample used.

Center for Educational Measurement, Inc.

The test of excellence

29

Recommendations
It should be noted that test scores are always subject to measurement errors. Hence, it is advised that other measures that assess other student characteristics (e.g., sociodemographics, motivation levels, etc.) may be considered before coming up with a final decision whether to accept or reject a candidate for admission to School A, B, or C. Test components should be reviewed and new test specifications should be considered to reflect subtests that highly correlate with academic performance
Center for Educational Measurement, Inc.

The test of excellence

30

Explore the option of restoring a national cut-off score for quality assurance Address the need for more research studies on the implications of NMAT on quality assurance. This will entail full cooperation among medical schools and other agencies in providing the necessary database.

Center for Educational Measurement, Inc.

The test of excellence

31