INDONESIAN 21ST CENTURY SKILLS SCALE FOR PROSPECTIVE SCHOOL COUNSELOR:

APPLICATION OF RASCH ANALYSIS

Ma’rifatin Indah Kholili, Universitas Sebelas Maret, Indonesia, https://orcid.org/0000-0003-1652-5859,

marifatin.ink@staff.uns.ac.id.
Novita Tri Hapsari, Universitas Sebelas Maret, Indonesia, novitatrihapsari@student.uns.ac.id
Adi Dewantoro, Universitas Sebelas Maret, Indonesia, adi_dewantoro@staff.uns.ac.id
Naharus Surur, Universitas Sebelas Maret, Indonesia, naharus67@staff.uns.ac.id

ABSTRACT
21st-century skills need to be mastered by every individual in this era. These skills are important as they
serve as the basis for developing an instrument to measure the level of 21st-century skills among prospective
school counselors. This study aims to develop the Indonesian 21st Century Skills Scales (ICSS) for prospective
school counselors. To this end, measurement development method was applied. The construct validity test
involved prospective school counselors (Guidance and Counseling students) Java, Sumatra, Kalimantan, and
Nusa Tenggara. Participants were 378 prospective school counselors, consisting of 308 female students and 70
male students. Data were collected by distributing the 21st-century skill scale through Google Form and
analyzed Using the Rasch Model with Winsteps software version 3.73. The Rasch analysis results provide
information about the quality of respondents and the instrument, the items that are easy and difficult for
respondents, the fit order of items, and the unidimensionality. Based on the validity test results, which refer to
the Mean Square Outfit (MNSQ), Standardized Outfit Z-Value (ZSTD), and Point Measure Correlation (Pt mean
Corr), five items were considered invalid, and hence removed from the instrument, leaving 72 valid items. The
item reliability data revealed that the person-item interaction (Cronbach's Alpha value) was categorized as very
good, with a result of .97, and the person reliability was .94. These results indicate good consistency between
respondents' answers and the instrument's items. The Mean Square Outfit results indicate that the actual data
were consistent with the requirements of the Rasch model. In other words, the 21st-century skills scale
instrument could be deemed accurate and precise. Future researchers are suggested to employ this instrument to
measure the 21st-century skills of guidance and counseling department students using experimental research
methods.

Keywords: pre-service counselor, rasch model, 21st century skills, scale validity

INTRODUCTION
21st-century skills are currently needed by many individuals worldwide, including students and
prospective school counselors. The 21st century are marked by globalization, which contributes to various
changes in economy, transportation, technology, communication, and information. The 21st century demands
various skills individuals should master. Therefore, education is expected to prepare students to acquire these
skills to achieve successful life (Tican & Deniz, 2019). In learning context, 21st-century skills also emphasize
one’s readiness to use knowledge for different purposes rather than merely memorizing facts (Rotherham &
Willingham, 2017).
Assessment and Teaching of 21st Century Skills (ATC21S) divides 21st-century skills into four
categories: way of thinking, way of working, tools for working, and skills for living in the world (Griffin &
Care, 2015). Way of thinking includes creativity, innovation, critical thinking, problem-solving, and decision-
making (Aslamiah et al., 2021). Way of working includes skills in communication, collaboration, and teamwork.
Tools for working include awareness as a global and local citizen, life and career development, and a sense of
personal and social responsibility. Skills for living in the world are based on information literacy, mastery of
new information and communication technologies, and the ability to learn and work through digital social
networks.
The 21st century skills are also relevant to the four pillars of learning, which include learning to know,
learning to do, learning to be, and learning to live together. By possessing 21st-century skills and competencies,
teachers will likely be able to facilitate students to achieve optimal self-development by utilizing technology
(Santos, 2017). The presence of 21st-century challenges should be properly offset by prospective counselors’
pivotal role and responsibility. (Kim et al., 2019). Prospective school counselors need to possess adequate
knowledge of technology, pedagogy, content, and understanding the interplay of these three aspects (Sadeck et
al., 2021).

Students in the 21st century have grown up in a fast-paced and easily accessible digital world, turning
them into technology-savvy individuals. Prospective school counselors, who will be responsible for preparing
students for the workforce, are expected to pay greater attention to students’ 21st-century skills(Boholano,
2017). 21st-century skills are helpful for the implementation of comprehensive school counseling programs
(Boulden, 2021).
The need for 21st-century skill mastery has not been followed by adequate measurement for all
educational levels. We have examined the development of 21st-century skill measurement in Indonesian context
and found two relevant studies. Khoiri (Khoiri et al., 2021) conducted a study to measure junior high school
students’ 21st-century skills, while Kelley (R. Kelley et al., 2019) conducted a study entitled “Factor analysis of
the development of 21st-century skill measurement for senior high school students.” These studies focused on
students at the secondary school level.
To date, no studies reported an instrument for measuring prospective school counselors’ 21st-century skill
in Indonesian context. The present study attempted to fill this gap by testing the validity and reliability of the
21st-century skill scale for prospective school counselors in Indonesia. To this end, Rasch Model analysis was
carried out to find out the fitness of the developed instrument.

Literature Review
Measurement
Latent individual skills could be measured when the measurement tool or instrument is available.
According to De Vellis (DeVellis, 2017), a measurement instrument is a collection of statement items used to
express the level of a theoretical variable that is not easily observed directly, often referred to as a ‘scale’.
Test items are devices used to gather information about the knowledge, skill, or task domains that defines the
construct. A construct is variable that can be defined and has strong characteristics (Haladyna & Rodriguez,
2013).
The development of instruments or measurement tools for assessment in counseling, psychology,
education, and other fields should be carried out within a general framework and provide theoretical
explanations about behavior and phenomena in these fields (Dimiter M. Dimitrov, 2012). The fundamental
consideration for counselors in measuring variables is to select the appropriate scale for specific functions. Thus,
a scale refers to one’s method to measure an individual construct (Foster, 2018).
To generate a well-founded psychological measurement scale or inventory, instrument developers need to
test data on a number of participants. The items are then examined to avoid cultural bias, and any items that may
be unfair or offensive to any group are eliminated (Foster, 2018). Among the important stages in instrument
development are validity and reliability tests. Validity is an integrated evaluation of an assessment to determine
the extent to which empirical evidence and theoretical rationales support the sufficiency and appropriateness of
conclusions and actions based on test scores or other assessment models (Dimiter M. Dimitrov, 2012).

METHOD
Research Design
This study utilized a measurement instrument development design. Data were collected through a survey. The
scale was developed following procedures outlined by DeVellis (DeVellis, 2017) :
1) Determine the aims of the measure: The first step was to determine what needs to be measured. It is
important to ensure that the measurement is grounded in substantive theory.
2) Generate an item pool: In the second step, we created the instrument. Items should reflect the
measurement purposes or be relevant to the construct being measured. It is also pivotal to consider the
number of item and the use of positively and negatively-worded statements.
3) Determine the format for measurement: The third step involved deciding the format of the scale.
 4) Have the initial item pool reviewed by experts: In this step, a group of knowledgeable individuals in
the content area review and evaluate the item pool to optimize the content validity of the scale.
5) Consider inclusion of validation items: The essence of the scale development is to ensure that the
developed item pool has been validated.
6) Administer items to a development sample: We administered the items into the scale after determining
the construct and their validity. The item management should consider the sample size.
7) Evaluate the items: After the initial item pool has been developed, tested, and administered to a
sufficiently large and representative sample, the next step was to evaluate the performance of each
item to identify fit items to form the scale.
8) Optimize scale length: In the final step, we decided set of items that demonstrate acceptable reliability
level.
Stages one to four were carried out in the previous year, and the present study continued previous studies by
conducting stages five to eight.
In this study, the scale analysis method follows the procedure presented by Sumintono (Sumintono, 2018),
namely:
(1) Verify the assumptions of unidimensionality and local independence of measurement
(2) Test the item-individual fit with the model Items with low fit values are excluded from further analysis.
Repeat the analysis until all items fit the model
(3) If the remaining number of items exceeds the targeted number, item selection can be performed based
on various considerations, such as: a) Items that do not overlap in their location with other items. b)
Items that can improve measurement reliability. c) Items with response options that follow a logical
sequence (assessing item characteristic curves).d) Items that provide relevant information for the
measurement function (assessing the information function graph).

Participants
Participants were prospective school counselors in Sumatra, Java, Kalimantan, and Nusa Tenggara Regions.
They were 21-24 years of age. Table 1 below presents the participants’ data.
Table 1. Research Participants
Description Number of Participants Percentages of
Participants(%)
Gender
Male 70 18.5
Female 308 81.5
Location
Java 277 73.28
Kalimantan 32 8.47
Sumatera 30 7.93
Nusa Tenggara 39 10.32

Data collecting instrument

Data were collected using the 21st century skill scale for school counselors. The scale was developed
based on the concept proposed by Assessment and Teaching of 21 st Century Skills (AT21CS), Partnership 21
(P21 Century Skills). It consisted of seventy-seven items, developed based on four aspects: (1) Ways of
Thinking (Learning and Innovation Skills), (2) Ways of Working (life and career skills), (3) Tools for Working
(Information, Media, and Technology Skills), and (4) Living in the World. Participants were asked to respond to
the items using the four-point likert scale: “strongly agree”, “agree”, “fairly agree”, “disagree”, and “strongly
disagree.”

Data collecting technique

Data were collected using the scale, which was distributed through Google Form, to find out prospective
school counselors’ 21st-century skill levels. The data collection process took three months, from August to
October 2022.
Data Analysis
Rasch Model analysis was conducted to test the accuracy and consistency of the 21st-century skills scale.
The Rasch Model was used because it provides a more comprehensive and informative depiction of the construct
being measured and the respondents (Winarti & Mubarak, 2019). In this quantitative data analysis, the Rasch
Model also tests the person and item fit simultaneously (Suryandai et al., 2022). Data were validated by
examining the accuracy of item responses, as indicated by Outfit Mean Square (MNSQ) values, where 0.5 <
MNSQ < 1.5. The accepted outfit Z standard (ZSTD) was -2.0 < ZSTD < +2.0, and the point-mean correlation
(Pt Mean Corr) was 0.4 < Pt Mean Corr < 0.85 (Suryandai et al., 2022)(Winarti & Mubarak, 2019).
If an instrument item meets at least one of the above criteria, the instrument item was considered
acceptable. The scale was deemed valid when all items met the fit criteria. The raw data collected on an ordinal
scale was converted into an interval scale using Rasch modeling with Winsteps software version 3.73 (Lee et al.,
2021).

RESULTS
Rasch Model Statistical Summary
The Rasch Model analysis results indicate that the 21st century skill scale developed in this study
exhibited a very good quality. From a sample of 378 respondents and a questionnaire consisting of 77 items, the
person measurement reliability was.94, while the item measurement reliability was .98. These values are above
the logit 0.0 and close to 1.0, indicating that respondents tended to agree with the statements about the items. A
summary of the statistical analysis results is presented in Figure 1.

Figure 1. Rasch Model Statistical Analysis Summary

The item reliability data showed that the person-item interaction (Cronbach's Alpha value) was
categorized as very good, with a result of .97, and the person reliability was .98. These results indicate good
consistency between respondents' answers and the instrument's items. The outfit mean square was 1.02, which
could be considered as an ideal value, i.e., 1.00. This result indicates that the actual data were consistent with the
requirements of the Rasch model. In other words, the 21st-century skills scale could be deemed accurate and
precise. The following table 2 presents the reliability.
 Table 2. Rasch Analysis Result

Psychometric Properties Person Item

N 378 77
Outfit mean square 1.02 1.02
Mean 1.31 .00
Standard Deviation 1.58 .79
Reliability .94 .98
Separation 4.08 7.07
Cronbach Alpha (KR-20) .97
Reliability

Person-Item Map
Rasch model analysis generated a comprehensive person-item map, also known as Wright Map, which is
presented in Figure 2. The data showed that seven participants had extreme maximum score, namely 121L,
131P, 132P, 136P, 213P, 272L and 321L, whose logit value was +7.

Figure 2. Wright Map (Person-Item Map)

Item Measure and Item Fit
The item logit measurement (item difficulty level) resulted in the highest logit value of 1.48 and the
lowest value of -1.90 with a standard deviation of 0.79. Based on the item logit measurement results, the
difficulty levels of the statement items could be categorized as follows: very difficult to understand, difficult to
understand, moderately easy to understand, easy to understand, and very easy to understand.
Item fit was used to assess the item fit with the model , determining whether or not the items function
normally in measurement. Several criteria were set to evaluate the item fit , including the Outfit Mean Square
(MNSQ), the Outfit Z-Standard (ZSTD), and the Point Measure Correlation (Pt Mean Corr).
Based on the item misfit data presented, 4 items did not meet the criteria for Outfit Mean Square
(MNSQ), 25 items did not meet the criteria for Outfit Z-Standard (ZSTD), and 5 items did not meet the criteria
for Point Measure Correlation (Pt Mean Corr). According to the data, there are 5 outlier or misfit items (i.e., P1,
P45, P47, P73, and P77) as they did not meet all three criteria. The remaining items that did not meet the outfit
z-standard criterion can still be retained. To conclude, five items were deemed invalid and hence removed from
the instrument, leaving seventy-two items in the 21st-century skills scale.

Differential Item Function

The differential item function (DIF) is used to detect the presence of bias in the instrument items. In
Rasch modeling, DIF is determined when individuals with the same latent trait level respond differently to an
item (Wright and Masters, 1982). Item bias occurs when individuals with certain characteristics are advantaged
compared to individuals with other characteristics. For example, when an item is easier for females than males,
then it is considered as gender bias.
Items that approach the upper limit have high difficulty levels, such as item number 46, while items that
approach the lower limit have low difficulty levels. The DIF test results, as shown in Figure 3, demonstrated
several biased items: items 1, 26, 46, 56, and 76, as they appeared to be easier for males (shown below) than
females. Two other biased items were 66 and 71, which appeared to be easier For female participants than male
participants. For other items, the difference in the ability to respond to the items was not significantly different.

PERSON DIF plot (DIF=$C83W1)

3 9 5 1 7 3 9 5 1 7 3
P1 P7 P1 P1 P2 P3 P3 P4 P4 P5 P6 P6 P7
1 7 13 19 25 31 37 43 49 55 61 67 73
2
1.5
DIF Measure (diff.)

1 L
0.5 P
0 *
-0.5
-1
-1.5
-2
-2.5
ITEM

Figure 3. Differential Item Function

Result
The 21st-century skill scale developed in this study comprises of four aspects: (1) Ways of Thinking (Learning
and Innovation Skills) consisting of five indicators; (2) Ways of ways of Working (life and career skills)
consisting of six indicators; (3) Tools for Working (Information, Media and Technology Skills) consisting of
three indicators,; and (4) Living in the World consisting of four indicators. The response choices were presented
in a Likert-scale, consisting of “strongly agree”, “Agree”, “Fairly Agree”, “Disagree”, and “Strongly Disagree.”
After undergoing the content validity (expert judgment) and construct validity tests, seventy-two items were
deemed valid and reliable. The distribution of aspect, indicator, and item examples are presented in Table 2.

Table 1. Aspect, Indicator, and Item Example of the 21st century skill scale

Reference Valid
Aspects Indicator Items Examples
s Item
Ways of a. Creativity and (Yudha et I think about innovation that aligns with 4
Thinking Innovation, al., 2018) my field of study.
(Learning and b. Critical (Changwo I will review and analyze every event 7
Innovation Thinking, ng et al., that occurs in life
Skills) 2018)
(Saleh,
2019)
c. Problem (Md, When encountering challenges, I set 3
Solving, 2019) strategies to overcome the challenges.
d. Decision (Mettas, I identify and analyze problems before 5
Making, 2011) making a decision
e. Learning to (Karatas & I learn about learning strategy to 3
Learn or Arpaci, optimize every opportunity.
Metacognitio 2021)
n (Krieger et
al., 2022)
Ways of a. Flexibility (Bayley, I can adapt to new tasks 4
Working (life and 2022)
and career Adaptability (Sulistyani
skills) ngsih &
Rahmawat
i, 2019)
b. Initiative and (Sulistyani I deliver new initiatives 2
Self-Direction ngsih &
Rahmawat
i, 2019)
c. Social and (Chiu et I learn the culture of place where I am 6
Cross-Cultural al., 2013) in.
Skills
d. Productivity (Sulistyani I make target for every activity. 4
and ngsih &
Accountabilit Rahmawat
y i, 2019)
e. Leadership (Sulistyani I can direct teams to achieve common 5
and ngsih & goals.
Responsibility Rahmawat
i, 2019)
f. Communicati (de I can express my ideas to my team. 4
on and Guzman &
Teamwork Choi,
2013)
Tools for a. Information (Virkus, I look for reliable sources of 2
Working Literacy 2018) information.
(Information, b. Media (Eristi & I optimize the technological 2
Media and Literacy Erdem, advancement
Technology 2017)
Skills), (HOBBS
& FROST,
2003)
 Reference Valid
Aspects Indicator Items Examples
s Item
c. General (Ben I can operate the digital technology as a 4
Knowledge Youssef et means to find out source of information
and ICT al., 2012)
(Information, (Bass,
Communicati 2007)
on and
Technology)
Literacy
Living in the a. Citizenship, (Mutch, I can process information and issues that 5
World. 2005) exist in the community
b. Personal and (Pozo et I have self-control in every condition 6
Social al., 2018) and situation.
Responsibility
,
c. Cultural (Perng & I understand that every individual has 2
Awareness Watson, different perceptions.
And 2012)
Competence (Gamst et
al., 2004)
d. Spiritual- (Vieten et I believe that success and failure 4
Religious al., 2013) depends on God's will
TOTAL VALID ITEMS 72

DISCUSSION
21st-century learning requires complex competencies, transformative technology-based learning, and non-
cognitive skills (Mutohhari et al., 2021). Educational curricula have focused on the importance of 21st-century
competencies, given the increasing urgency of skills in learning. Factors including globalization and
technological advancements have prompted educational planners to consider the suitability of their school
systems to ensure that their students possess the required skills when entering the workforce in order to leverage
future opportunities. Based on a study by Toward Universal Learning, there are six measurement areas as
important learning opportunities for children and adolescents to succeed in a global society. These six areas
represent how learning should be measured globally, and one of them is "An adaptable, flexible skill set to meet
the demands of the 21st century." Skill measurement is one of the crucial areas, yet the development of
instruments is still limited (Laajaj & Macours, 2019).
School counselors also play an active role in honing 21st-century skills for students. Personal/social skills,
academic skills, and career transition skills contribute to supporting individuals' abilities to face challenges in the
21st century era (Yuksel-Sahina, 2012)(Kulcsár et al., 2020). The prospective counselors’ skills encompass not
only internal skills such as critical thinking, empathy, and positive attitudes but also skills applicable to the social
sphere and the way of life in the world, especially in helping students understand themselves and the demands of
the time development (Mcmahon et al., 2016)(Alavi et al., 2012). It is essential for prospective counselors to
enhance their 21st-century skills to meet the rapidly changing society through their work in schools, enabling
them to respond to the evolving demands of the community (Indreswari et al., 2020).
The implications of the study include the understanding that 21st-century skills encompass more complex
aspects, such as way of thinking, way of working, tools for working, and skills for living in the world.
Therefore, 21st-century skills can be combined with other relevant skills to have a broader scope. Additionally,
specific aspects are analyzed simultaneously and calibrated on a single linear scale (i.e., the logit scale).
Simultaneous calibration allows the measurement of seriousness level of a scale with regard to the test takers’
capacity, difficulty, and the criteria .
Validity test was conducted by considering MNSQ, fit of standard outfit Z, and point-mean correlation,
resulting in data stating that the 21st-century skills scale is valid, accurate, and consistent. From the presentation
of the item data on infit and outfit, five items did not meet the criteria and hence removed, leaving 72 items for
measuring 21st-century skills. Infit and outfit values are related to the chi-square measure. Lower values
indicate overly structured responses. Meanwhile, higher values indicate overly random responses (i.e., less
discriminative), and thus, they do not need to be included in the instrument (Willse, 2017). The RASCH method
can also identify biased items related to the item difficulty level when responded by males or females based on
DIF analysis. Differential Item Functioning (DIF) analysis is a statistical technique used to ensure the equity of
assessment instruments. DIF analysis plays a crucial role in ensuring the equality and fairness of educational
assessments because tests that are free from DIF are considered equitable and fair for all participants (Wyse &
Mapuranga, 2009)(Myers et al., 2006).

CONCLUSION
The study purpose of developing the Indonesian 21st Century Skills Scale (ICSS) for school counselors has been
achieved. The instrument has been tested for validity and reliability based on user validity test. This study
yielded 72 valid and reliable items that are ready to be used by prospective school counselors.
The limitation of this research is that it was only tested on prospective school counselors and has not been tested
on students in other educational fields. Therefore, future researchers are recommended to test the validity of this
instrument in other educational fields. Furthermore, the instrument test should include students in higher
education. Future researchers could also expand the development of the 21st-century skill scale for students at
the secondary education level.

Recommendation
Future studies need to clarify and reaffirm the generalizability of this study finding with respect to the
measurement of 21st-century skill measurement among guidance and counseling department student. Guidance
and counseling department students should be aware about 21st-century skill measurement and focus on
developing those skills. Lecturers in guidance and counseling department need to consistently facilitate
classroom activities to optimize the development of 21st-century skills. Future researchers are suggested to
employ this instrument to measure the 21st-century skills of guidance and counseling department students using
experimental research methods.

Limitation
This study has several limitations, including being conducted only among university students. Therefore, the
results cannot be generalized to other education levels. Future researchers may consider collecting more data
with a broader scope across different regions of Indonesia.
Additionally, this study focused only on guidance and counseling. Hence, future researchers may consider
developing measurement tools for prospective teachers in other fields (such as language education, arts,
mathematics, natural sciences, or others) with similar variables. Future studies may apply experimental design
to test this scale.

ACKNOWLEDGMENTS
We would like to thank the Institute of Research and Community Service of Sebelas Maret University for
funding this research through Dana Hibah Penelitian PNBP. We would also thank the Guidance and Counseling
Department of Sebelas Maret University, Surakarta, Indonesia.

REFERENCES

Alavi, M., Boujarian, N., & Ninggal, M. T. (2012). The Challenges of High School Counselors in Work Place.
Procedia - Social and Behavioral Sciences, 46, 4786–4792. https://doi.org/10.1016/j.sbspro.2012.06.335
Aslamiah, A., Abbas, E. W., & Mutiani, M. (2021). 21st-Century Skills and Social Studies Education. The
Innovation of Social Studies Journal, 2(2), 82. https://doi.org/10.20527/iis.v2i2.3066
Bass, J. M. (2007). A developmental model of basic ICT skills for pre-service trainee teachers. 2007
International Conference on Information and Communication Technologies and Development, ICTD 2007,
December 2007. https://doi.org/10.1109/ICTD.2007.4937404
Bayley, S. H. (2022). Learning for adaptation and 21st-century skills: Evidence of pupils’ flexibility in Rwandan
primary schools. International Journal of Educational Development, 93(July), 102642.
https://doi.org/10.1016/j.ijedudev.2022.102642
Ben Youssef, A., Dahmani, M., & Omrani, N. (2012). Students’ E-Skills, Organizational Change and Diversity
of Learning Process: Evidence from French Universities in 2010. SSRN Electronic Journal, 12.
https://doi.org/10.2139/ssrn.2065366
Boholano, H. (2017). Smart social networking: 21st Century teaching and learning skills. Research in Pedagogy,
7(2), 21–29. https://doi.org/10.17810/2015.45
Boulden, R. (2021). Developing 21st-Century Skills through Restorative Practices. The Clearing House: A
Journal of Educational Strategies, Issues and Ideas, 94(5), 205–215.
https://doi.org/10.1080/00098655.2021.1915740
Changwong, K., Sukkamart, A., & Sisan, B. (2018). Critical thinking skill development: Analysis of a new
learning management model for Thai high schools. Journal of International Studies, 11(2), 37–48.
https://doi.org/10.14254/2071-8330.2018/11-2/3
Chiu, C. Y., Lonner, W. J., Matsumoto, D., & Ward, C. (2013). Cross-Cultural Competence: Theory, Research,
and Application. Journal of Cross-Cultural Psychology, 44(6), 843–848.
https://doi.org/10.1177/0022022113493716
De Guzman, A. B., & Choi, K. O. (2013). The relations of employability skills to career adaptability among
technical school students. Journal of Vocational Behavior, 82(3), 199–207.
https://doi.org/10.1016/j.jvb.2013.01.009
DeVellis, R. F. (2017). Scale Development Theory and Applications (4th Edition) (Vol. 5, Issue 3).
https://doi.org/Doi: 300.72—dc23 LC
Dimiter M. Dimitrov. (2012). Statistical Methods for Assessment Scale Data Validation of in Counseling and
Related Fields. In American Counseling Association. https://doi.org/10.1088/1751-8113/44/8/085201
Eristi, B., & Erdem, C. (2017). Development of a Media Literacy Skills Scale. Contemporary Educational
Technology, 8(3), 249–267. https://doi.org/10.30935/cedtech/6199
Foster, L. H. (2018). Assessment in Counseling. In Introduction to the Counseling Profession.
https://doi.org/10.4324/9781315537061-10
Gamst, G., Dana, R. H., Der-Karabetian, A., Aragon, M., Arellano, L., Morrow, G., & Martenson, L. (2004).
Cultural competency revised: The California Brief Multicultural Competence Scale. Measurement and
Evaluation in Counseling and Development, 37(3), 163–183.
https://doi.org/10.1080/07481756.2004.11909758
Griffin, P., & Care, E. (2015). Assessment and Teaching of 21st Century Skills Methods and Approach. In
Assessment and Teaching of 21st Century Skills. https://doi.org/10.1007/978-94-017-9395-7_15
Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Developing and Validating
Test Items, 1–446. https://doi.org/10.4324/9780203850381
HOBBS, R., & FROST, R. (2003). Measuring the acquisition of media-literacy skills. Reading Research
Quarterly, 38(3), 330–355. https://doi.org/10.1598/rrq.38.3.2
Indreswari, H., Prihatmoko, Y., & Nindigraha, N. (2020). Candidate for Personal Learning Counselors
Electronic Materials on Learning Environment. 508(Icite), 568–572.
https://doi.org/10.2991/assehr.k.201214.298
Karatas, K., & Arpaci, I. (2021). The role of self-directed learning, metacognition, and 21st century skills
predicting the readiness for online learning. Contemporary Educational Technology, 13(3).
https://doi.org/10.30935/cedtech/10786
Khoiri, A., Evalina, Komariah, N., Utami, R. T., Paramarta, V., Siswandi, Janudin, & Sunarsi, D. (2021). 4Cs
Analysis of 21st Century Skills-Based School Areas. Journal of Physics: Conference Series, 1764(1).
https://doi.org/10.1088/1742-6596/1764/1/012142
Kim, S., Raza, M., & Seidman, E. (2019). Improving 21st-century teaching skills: The key to effective 21st-
century learners. Research in Comparative and International Education, 14(1), 99–117.
https://doi.org/10.1177/1745499919829214
Krieger, F., Azevedo, R., Graesser, A. C., & Greiff, S. (2022). Introduction to the special issue: the role of
metacognition in complex skills - spotlights on problem solving, collaboration, and self-regulated learning.
Metacognition and Learning, 17(3), 683–690. https://doi.org/10.1007/s11409-022-09327-6
Kulcsár, V., Dobrean, A., & Gati, I. (2020). Challenges and difficulties in career decision making: Their causes,
and their effects on the process and the decision. Journal of Vocational Behavior, 116(September 2019),
103346. https://doi.org/10.1016/j.jvb.2019.103346
Laajaj, R., & Macours, K. (2019). Measuring Skills in Developing Countries. Journal of Human Resources,
56(4), 1–46. https://doi.org/10.3368/jhr.56.4.1018-9805R1
Lee, W. L., Chinna, K., & Sumintono, B. (2021). Psychometrics assessment of HeartQoL questionnaire : A
Rasch analysis. 1–5. https://doi.org/10.1177/2047487320902322
Mcmahon, H. G., Paisley, P. O., & Ed, D. (2016). School Counseling for the 21st century: Challenges and
Opportunities. January 2001, 116–125.
Md, M. R. (2019). 21st Century Skill “Problem Solving”: Defining the Concept. Asian Journal of
 Interdisciplinary Research, 2(1), 64–74. https://doi.org/10.34256/ajir1917
Mettas, A. (2011). The development of decision-making skills. Eurasia Journal of Mathematics, Science and
Technology Education, 7(1), 63–73. https://doi.org/10.12973/ejmste/75180
Mutch, C. (2005). Citizenship education: does it have a place in the curriculum? Curriculum Matters, 1, 49–70.
https://doi.org/10.18296/cm.0069
Mutohhari, F., Sutiman, S., Nurtanto, M., Kholifah, N., & Samsudin, A. (2021). Difficulties in implementing
21st century skills competence in vocational education learning. International Journal of Evaluation and
Research in Education, 10(4), 1229–1236. https://doi.org/10.11591/ijere.v10i4.22028
Myers, N. D., Wolfe, E. W., Feltz, D. L., & Penfield, R. D. (2006). Identifying differential item functioning of
rating scale items with the Rasch model: An introduction and an application. Measurement in Physical
Education and Exercise Science, 10(4), 215–240. https://doi.org/10.1207/s15327841mpee1004_1
Perng, S. J., & Watson, R. (2012). Construct validation of the Nurse Cultural Competence Scale: A hierarchy of
abilities. Journal of Clinical Nursing, 21(11–12), 1678–1684. https://doi.org/10.1111/j.1365-
2702.2011.03933.x
Pozo, P., Grao-Cruces, A., & Pérez-Ordás, R. (2018). Teaching personal and social responsibility model-based
programmes in physical education: A systematic review. European Physical Education Review, 24(1), 56–
75. https://doi.org/10.1177/1356336X16664749
R. Kelley, T., Geoff Knowles, J., Han, J., & Sung, E. (2019). Creating a 21st Century Skills Survey Instrument
for High School Students. American Journal of Educational Research, 7(8), 583–590.
https://doi.org/10.12691/education-7-8-7
Rotherham, A. J., & Willingham, D. T. (2017). The 21st Century Skills. Key Competencies and Contemporary
Skill Development in Education, 562–1065. https://doi.org/10.4018/978-1-4666-9823-9.les2
Sadeck, O., Moyo, M., Tunjera, N., & Chigona, A. (2021). Pre-service teacher training for realities of 21st
century classrooms. 8th International Conference on Educational Technologies 2021, ICEduTech 2021
and 17th International Conference on Mobile Learning 2021, ML 2021, March 2020, 74–82.
https://doi.org/10.33965/ml_icedutech2021_202102l010
Saleh, S. E. (2019). CRITICAL THINKING AS A 21 st CENTURY SKILL: CONCEPTIONS,
IMPLEMENTATION AND CHALLENGES IN THE EFL CLASSROOM. European Journal of Foreign
Language Teaching, 4(1), 1–16. https://doi.org/10.5281/zenodo.2542838
Santos, J. M. (2017). 21st Century Learning Skills: A Challenge in Every Classroom. Ijemr, 1(1), 31–35.
https://doi.org/10.22662/ijemr.2017.1.1.031
Sulistyaningsih, S., & Rahmawati, V. E. (2019). The Implementation of 21st Century Skills as the New Learning
Paradigm to the Result of Student’s Career and Life Skills. JournEEL (Journal of English Education and
Literature), 1(1), 1–10. https://doi.org/10.51836/journeel.v1i1.37
Sumintono, B. (2018). Rasch Model Measurements as Tools in Assesment for Learning. October 2017.
https://doi.org/10.2991/icei-17.2018.11
Suryandai, K. C., Rokhmaniyah, Salimi, M., & Fatimah, S. (2022). Involvement of Teachers, Parents, and
School Committees in Improving Scientific Attitudes of Elementary School Students: Application of
Rasch Model Analysis. International Journal of Educational Methodology, 8(4), 783–794.
https://doi.org/10.12973/ijem.8.4.783
Tican, C., & Deniz, S. (2019). Pre-service teachers’ opinions about the use of 21st century learner and 21st
century teacher skills. European Journal of Educational Research, 8(1), 181–197.
https://doi.org/10.12973/eu-jer.8.1.181
Vieten, C., Scammell, S., Pilato, R., Ammondson, I., Pargament, K. I., & Lukoff, D. (2013). Spiritual and
religious competencies for psychologists. Psychology of Religion and Spirituality, 5(3), 129–144.
https://doi.org/10.1037/a0032699
Virkus, S. (2018). Information literacy and learning. Libraries Without Walls 5, January, 97–108.
https://doi.org/10.29085/9781856047876.010
Willse, J. T. (2017). Polytomous rasch models in counseling assessment. Measurement and Evaluation in
Counseling and Development, 50(4), 248–255. https://doi.org/10.1080/07481756.2017.1362656
Winarti, A., & Mubarak, A. (2019). Rasch Modeling: A Multiple Choice Chemistry Test. Indonesian Journal on
Learning and Advanced Education (IJOLAE), 2(1), 1–9. https://doi.org/10.23917/ijolae.v2i1.8985
Wyse, A. E., & Mapuranga, R. (2009). Differential Item Functioning Analysis Using Rasch Item Information
Functions. International Journal of Testing, 9(4), 333–357. https://doi.org/10.1080/15305050903352040
Yudha, F., Dafik, D., & Yuliati, N. (2018). The Analysis of Creative and Innovative Thinking Skills of the 21st
Century Students in Solving the Problems of “Locating Dominating Set” in Research Based Learning.
International Journal of Advanced Engineering Research and Science, 5(3), 163–176.
https://doi.org/10.22161/ijaers.5.3.21
Yuksel-Sahina, F. (2012). School Counselors’ Assessment of the Psychological Counseling and Guidance
Services they Offer at their Schools. Procedia - Social and Behavioral Sciences, 47, 327–339.
https://doi.org/10.1016/j.sbspro.2012.06.659