1

The Effects of Teaching Critical Thinking Skills on Standardized Test Performance

Nai C. Saelee

Touro University California

 2

Abstract

The proposed research study explores whether there is a relationship between developing critical
thinking skills and standardized test performance. The 21st century has called for a change in
our educational system, shifting it from traditional rote learning to a necessity for critical
thinking skills. Various academically high scoring countries and states have already shifted their
curriculum to reflect 21st century learning. However, it is still unclear whether teaching critical
thinking will lead to increased test performance. This research study will employ a mixed design
of a quasi-experimental and naturalistic approach using a treatment and control group.
Quantitative data will be collected using a pre- and post-test (taken from the CAASPP practice
test) as well as a student self-assessment while qualitative data will be collected in the form of a
metacognitive journal. Samples include two demographically similar groups (each totaling 30
students) in which the control group is chosen randomly. Samples include females and males,
English Learners, and students with low socio-economic and diverse ethnic backgrounds. The
sample will not include students with disabilities. There is no potential risk to human subjects
because the research will be conducted in a normal educational setting using normal practices.
Benefits are unknown. Data will be collected and stored in a secure place and recorded and
saved in an excel spreadsheet on a password protected personal computer. The research study
will be conducted at American Canyon Middle School in the Napa Valley Unified School
District over a two week period in Spring 2017.

Keywords: critical thinking skills, standardized test performance, metacognition, conceptual

understanding and procedural fluency
 3

The Effects of Teaching Critical Thinking Skills on Standardized Test Performance

In a world where the average adult makes 35,000 decisions a day in contrast to the 3,000

that of a child, one could infer that decision-making is a necessary and essential skill to

successfully function in society. To be more exact, the ability to make wise decisions is integral

to academic, professional, and personal success. Also, to put it into better context, the average

person is bombarded with a massive amount of choices, choices which all carry certain

consequencesbig and small as well as good and bad. Over time these choices compound, they

accumulate together over a lifetime and impact the individual as well as their spouses, families,

teams, businesses, organizations, communities, states, nations, and even the world. Furthermore,

successfully functioning in society now requires the new generation to embrace the modern

information agecharacterized by advancements in technology and a demand for critical

thinking.

The 21st century has radically changed how the game of life is played. With the

constant stream of information that the individual encounters on a daily basis, how does one

discern what information is relevant, reliable, and important? In 2017, now more than ever, it

has become more challenging to know who and what to trust amid the influx of varying quality

of information that is integrated with information referred to as fake news and alternative

facts. Scarier is the fact that much of the information or fake news comes from traditionally

trusted authorities, such as the president of the United States and other high office government

officials. Not only will the new generation be expected to sift through a wealth of information

that is offered to them by the advancements of the 21st century, they will need critical thinking

abilities to help them decide what to do and what to think with that information. As the

impending generation responsible for running the country, they are tasked with electing
 4

government officials, voting on ballot initiatives such as state and local taxes, and affecting

change in many other ways.

Finally, the new generation must also prepare to enter the professional world which

demands specialized skills and knowledgein other words, critical thinking skills (Darling-

Hammond, 2009). Decision-making permeates all facets of life and the workplace is no

exception. A survey done by the Association of American Colleges and Universities (AACU)

indicates that 93% of business and non-profit leaders believe that a demonstrated capacity to

think critically, communicate clearly, and solve complex problems is more important than [an]

undergraduate major and more than 75% of those surveyed stated that they want more emphasis

on critical thinking, complex problem solving, written and oral communication, and applied

knowledge in real-world settings for all colleges and universities (Su, Ricci, & Mnaatsakanian,

2016).

According to an article from CNBC.com, a company called CareerCast evaluated data

gathered from the US Bureau of Labor Statistics, the Census Bureau, trade association studies,

and other sources to determine the best and worst jobs of 2015 (OBrien, 2015) . The results

showed that the list of the top 10 jobs was dominated by occupations in the field of STEM

(Science, Technology, Engineering, and Math), a list that had traditionally been occupied by

health care jobs. STEM jobs accounted for 15.1% of the professional labor force and 5.9% of

the total U.S. workforce. To add to this statistic, from 2005 to 2015, computer and mathematical

related occupations accounted for 79.5% of job growth in STEM occupations. In summary,

STEM jobs, specifically in the field of mathematics, are on the rise making the ability to problem

solve crucial.

Now that the trend in the job market and the importance of STEM has been established, it
 5

is also important to note that prospective U.S. STEM workers are threatened by offshoring and

immigration policies. More than ever, candidates vying for employment are in a global

competition. According to the Department for Professional Employees (DPE), skilled guest

workers made up a very small percentage of the overall U.S. workforce but were

disproportionately concentrated in STEM industries. The DPE also indicated that the offshoring

and outsourcing is not influenced by a qualified workforce shortage but rather by financial

factors. One solution to this problem might be to prepare the new generation to be critical

thinkers (creative and innovative thinkers) such that they are the next Steve Jobs: invaluable,

irreplaceable, and an asset to the companys progress and success.

Synthesizing the information above, the implication is that it is imperative for the

education system to shift towards a reform that better arms students with the 21st century tools

for making informed decisions, minimizing prejudice and bias, maximizing understanding, and

contributing wisely and effectively to a global society as well as an emphasis on sufficiently

preparing students in mathematics and science (Darling-Hammond, 2013; Paul & Elder, 2006).

That begs the question, where is the United States in terms of preparing their students for this

outcome?

Problem Statement

International Context. Relative to the rest of the world, the U.S. is still severely lagging

behind the top achieving countries. According to the Trends in Mathematics and Science Study

(TIMSS 2015), eighth grade students have, on average, shown a gradual but long-term

improvement in mathematics scores since 1995 to 2015, from 492 to 518 points

(https://nces.ed.gov/pubs2017/2017002.pdf). Although this gain is a step in the right direction,

data shows that the top five achieving countries are still outperforming the rest of the world
 6

considerably. The difference between U.S. scores (518 points) and the top performing country,

Singapore (621 points), was 103 points. In TIMSS 2015, there are four levels of student

achievement, referred to as international benchmarks: Advanced, High, Intermediate, and Low.

Students at the advanced level demonstrate the capacity for critical thinking. The data indicates

that merely 10% of U.S. eighth graders reached the advanced international benchmark in

mathematics compared to 55% of students in Singapore. Statistical evidence from another

international assessment, Programme International Student Assessment (PISA 2012),

coordinated by the Organization for Economic Cooperation and Development (OECD) offers

similar data to TIMSS. PISA reports mathematics literacy in terms of six proficiency levels in

which five and above are considered to be top performers who demonstrate advanced

mathematical thinking and reasoning required to extrapolate from what is learned and apply that

knowledge in unfamiliar settings to solve problems of greater complexity. The percentage of top

performers who have shown proficiency in critical thinking in the U.S. was lower than the

average of all 34 participating OECD. Only 9% of U.S. students reached level five proficiency

compared to the OECD average of 13%. Finally, of 34 participating OECD countries, the U.S.

ranked 27 near the very bottom.

National Context. Nationally, data assessment from The National Assessment of

Educational Progress (NAEP 2015) shows that only 33% of U.S. students in the eighth grade are

performing at or above the proficient level in mathematics

(https://www.nationsreportcard.gov/reading_math_2015/#mathematics/acl?grade=8). That is,

67% of students (or more than half of the nations eighth graders) are not meeting the standards.

Data also shows that the average mathematics scores of the nation have dropped from a rate of

35% proficiency to 33% proficiency. While this statistics can be disheartening, it is important to
 7

note that since the implementation of Common Core in 2010, scores have in fact increased but

only by one or two points within a span of five years. The gradual but steady growth is

encouraging.

State Context. Based on standardized test scores reported by NAEP 2015, California

ranked 41 out of 53 states and jurisdictions with an average rate of proficiency in mathematics

(across all grades) at 27% which is below the national average of 33%

(https://www.nationsreportcard.gov/reading_math_2015/#mathematics/state/comparisons/NP?gr

ade=8). In response to the poor results of state, national and international assessment data,

California has adopted the Common Core State Standards (CCSS) and joined with the

Partnership for 21st Century State Leadership Network (P21) to promote real-world problem

solving and college and career readiness. California has also worked hard to incorporate

Science, Technology, Engineering, and Mathematics (STEM) within the educational school

system to address its evident deficiencies.

Local Context. At the district site for this study, there is positive movement and change.

Comparing state, district, and school site data from the California Assessment of Student

Performance and Progress (CAASPP 2015 & 2016), the average proficiency rate among eighth

grade students statewide was 33% in 2015 increasing to 36% in 2016, district wide the rate

remained at 32% in both 2015 and 2016, and for one school site in the district, the rate was 33%

in 2015 increasing to 38% in 2016 (http://www.cde.ca.gov/nr/ne/yr16/yr16rel57.asp). The

district overall performed below the state average but the individual school site exceeded the

district, state, and national average rate of proficiency. The evidence supports that the individual

school site and district seem to have accounted for the demands of the 21st century job market

and is using local, state, national, and international assessment data to inform its curriculum
 8

planning. It is apparent that this school district is working towards preparing their students for a

competitive world, beyond the classroom.

One key reform that has shown some success is the implementation of 21st century

learning and the 6 Cscommunication, collaboration, critical thinking, creativity, character,

and global citizenship. One way 21st century learning and teaching is incorporated into the

school curriculum is through Problem Based Learning (PrBL). PrBL is a teaching approach that

incorporates inquiry learning. Students investigate an authentic, engaging, and complex problem

or question, and create a product that is presented to an authentic audience beyond their

classroom.

Importance of the Problem

The problem with the curriculum that incorporates PrBL is that it is still relatively unclear

as to what kinds of teacher knowledge or behavior are necessary to teach critical thinking skills.

Integrating an approach that includes authenticity, rigor, and motivation does not ensure that

critical thinking skills are being taught. To be sure, critical thinking skills are necessary to

complete the task, but the issue is determining whether or not there is a transfer of critical

thinking skills to the students. Research supports this claim, arguing that although critical

thinking skills are valued, they are seldom explicitly taught to students; they go on to explain

how instructors display the product of these skills but students rarely witness the processes by

which their teachers analyze texts, compare conflicting interpretations, or discover patterns in

seemingly random information and evidence (Kurfiss, 1988). This is precisely the case with

PrBL, students are tasked to utilize critical thinking skills to problem solve but are not explicitly

taught the requisite skills to complete these tasks and because critical thinking skills are not

explicitly nor systematically being taught, it is unclear as to what extent students are developing
 9

critical thinking skills as well as how it would affect their standardized test performance.

Purpose of the Study

The purpose of this study is to investigate how teaching metacognitive strategies, as a

critical thinking skill, will impact student performance on standardized tests in the mathematics

content area at the secondary education level. This study is designed to define critical thinking

skills and the requisite skills of a critical thinker in the mathematics content area. The study will

also determine the effectiveness of teaching strategies in developing critical thinking skills at the

secondary education level. Students will be taught using three strategies: (a) explicit instruction

using a think aloud, (b) metacognitive regulation strategies through the use of metacognitive

scaffolding and metacognitive journaling, and (c) use of feedback through Socratic Questioning

and Socratic Seminars. As a result of this study, it is hopeful that there will be an increase in

critical thinking skills and hence, higher student achievement on standardized tests.

Significance of the Study

The single most powerful systemic force in motivating change is assessment, put

eloquently, What is tested is what gets taught (Schoenfeld, 1992, p. 69). Since the 1970s,

tests have shifted yet again from an emphasis on rote memory to assessing critical thinking and

so it follows that curriculum and classroom instruction must also adapt. With the adoption of the

Eight Mathematical Practices embedded in the CCSSM, teachers around the country are seeking

ways to align their instruction to these standards. In her research described in detail below in the

instructional strategies section, Conrady (2015) stressed that the development of mathematical

practices requires a deep understanding of both content and underlying thinking processes. Like

other researchers, she emphasized that these behind the scenes and often time, messy thinking

processes are rarely shared or discussed in the classroom. A prominent problem in classrooms
 10

across the U.S. is when students come up against cognitively complex tasks and mathematical

problem solving that require the use of critical thinking skills, students struggle because they do

not have the ability to identify when specific content knowledge should be applied, fail to

evaluate their strategys effectiveness, or give up. Thus, it is important that teachers teach how

to think about math rather than memorizing formulas (Su, Ricci, & Mnaatsakanian, 2016).

Moreover, the consensus among the mathematics education community is that although critical

thinking has been described as a desirable proficiency, there have only been sporadic studies on

its inclusion in curriculum and most of the previous studies are at a higher education level and in

many cases, a non-mathematics field (Kurfiss, 1988).

Literature Review

Critical Thinking

The concept of critical thinking is not a new innovation. Critical thinking has existed as

early as the time of Socrates (c. 470399 BC) more than 2,400 years ago as a deep questioning

technique, a method of disciplined and rigorous questioning geared towards the logical analysis

and evaluation of the reliability and validity of beliefs (Paul & Elder, 2014). However, scholars

such as John Dewey, Robert Ennis, Richard Paul, and many other researchers gradually began to

expand on this early notion of critical thinking which has now come to include reflective

thinking, creative thinking, problem-solving, and metacognition (Dewey, 1910; Ennis, 1985;

Paul & Elder, 2006).

Critical Thinking Skills. Critical thinking skills include the ability to think reflectively

and reasonably, analyze arguments, challenge assumptions, discern opinion from fact, evaluate

issues from various perspectives, problem solve using flexible thinking, and self-regulate or

think metacognitively (Ennis, 1985; Paul & Elder 2006). All critical thinking skills are not
 11

created equal, more precisely, depending on the discipline, some critical thinking skills may be

valued more highly than others. For example, in an English course there may be an emphasis on

the ability to analyze and evaluate arguments and assumptions whereas skills such as problem

solving and self-regulation may be considered more essential in mathematics. In fact, Paul and

Elder (2006) frame the entire act of critical thinking as a metacognitive process, suggesting that

just becoming aware of and thinking about ones own thinking can improve a persons ability to

evaluate arguments and challenge assumptions. Within the context of mathematics, critical

thinking skills are described as the ability to problem solve, utilizing flexibility of thinking and a

broad repertoire of techniques for dealing with non-routine problems along with the ability to

reflect on progress as well as analyze and interpret vast amounts of data (Schoenfeld, 1992).

Metacognition

Metacognition is defined as the awareness and regulation of ones own thought processes

(Flavell, 1979). To elaborate, metacognition is comprised of two components: metacognitive

knowledge and metacognitive regulation. Metacognitive knowledge consists of three types of

knowledge: declarative knowledge, procedural knowledge, and conditional knowledge.

Declarative knowledge is what one knows, procedural knowledge is how one applies that

knowledge, and conditional knowledge is the when, where, and why one uses that knowledge.

Metacognitive regulation consists of three elements: planning, monitoring, and evaluating

(Schraw & Moshman, 1995). Knowledge and regulation are both vital for problem solving.

Intersection of Metacognition and Problem Solving. Mathematicians have long since

recognized the close relationship that exists between these two critical thinking skills,

metacognition and problem solving (Gray, 1991). Metacognition and problem solving often

have overlapping elements. During problem solving, metacognitive skills are utilized to analyze
 12

information, identify goals within the task, plan, monitor progress, consider alternatives, and

evaluate decisions and outcomes of a problem (Garofalo & Lester, 1985, Polya, 1957).

Significance of Metacognition. Metacognition can play a crucial role in learning and

increase conceptual understanding and procedural fluency (Jbeili, 2012). Because learning shifts

from conscious to automatic processing (concept mastery--conceptual understanding and

procedural fluency), it is important that increased consciousness is paid during the first stage of

learning for it to lead to deeper knowledge (Pammu, Amir, and Maasum, 2011). However,

acquiring conceptual understanding and procedural fluency does not ensure that the individual

will know how or when to apply it to more complex or unfamiliar problems (Kurfiss, 1988;

Mayer, 1993). Studies show that in many cases students have the necessary knowledge base

needed to solve a problem but lack the metacognitive knowledge known as conditional

knowledge to help them determine how to utilize that knowledge base. Unless the world is a

game of Jeopardy, there is no advantage to possessing great amounts of knowledge that cannot

be utilized. Moreover, studies indicate that the presence of strong metacognitive skills is a better

predictor of student problem solving success than their aptitude and can even compensate for low

aptitude (Schraw & Moshman, 1995; Swanson, 1990). Hence, it is important to help students

develop strong metacognitive skills.

Research-Based Instructional Strategies for Developing Metacognitive Skills

Extensive research has been done detailing the value of various instructional strategies

for developing metacognitive thinking skills. These strategies include: (a) direct explicit

instruction, often in the form of a think aloud, modeling, and whole class discussion, (b)

metacognitive regulation strategies that helps students plan, monitor, and evaluate their learning,

also accomplished using think alouds as well as journaling in order to discuss and document
 13

thinking, and (c) teacher and peer feedback, achieved through teacher questioning and prompting

in the form of Socratic Questioning and peer interaction in the form of Socratic Seminars.

Several studies show that metacognitive regulation and feedback typically go hand in hand and

are directly supported by explicit instruction. In both cases, explicit instruction is the means for

teaching these skills. Moreover, it is important to note that numerous studies incorporate similar

metacognitive regulation strategies, often, referred to by many different names: comprehension

monitoring, metacognitive training, and metacognitive scaffolding.

Explicit Instruction. Teachers expect students to utilize critical thinking skills to

produce high quality work for class projects but how can students or anyone for that matter, be

expected to apply skills they have never been effectively or explicitly been taught? How can

they know what they just learned if they are not told what it is they learned? How can they know

how, when, or why to use it if they were not shown? Studies show that explicit instruction of

metacognitive thinking is beneficial to students, especially struggling students because it offers

them flexibility, greater efficiency, and transferability of skills to unfamiliar situations during

problem solving (Lin, 2001; Pintrich, 2002; Schoenfeld, 1992). These skills are not obtained

automatically, thus, it is important that teachers help students develop metacognitive thinking

with the use of direct and explicit instruction, making the thinking process visible (Conrady,

2015). To achieve this, students must be given opportunities to develop their metacognitive

knowledge and regulation skills these are the skills that allow a student to know when and why

a certain procedure should be applied; some scaffolding activities that allow students to practice

these skills are whole class discussions, modeling of the problem solving process, a think aloud,

[Socratic] questioning, writing about thinking, prompting, using sentence starters, and explicit

instruction about thinking and metacognition (Flavell, 1979; Goos, et al., 2002; Pintrich, 2002).
 14

Conrady (2015) conducted a naturalistic inquiry study investigating explicit modeling of

metacognitive thinking embedded in two university level geometry courses for pre-service

education teachers. In the study, 51 students were exposed to explicit instruction of

metacognitive thinking strategies such as modeling, think alouds, prompting, and questioning.

Observations on the explicit thinking of the students were documented through field notes and

transcribed videotapes. Conrady (2015) concluded that the use of think alouds and modeling

helped students develop and make their own procedural thinking explicit. However, she also

found that students did not demonstrate independent regulatory thoughts during problem solving

and that most regulatory thoughts required prompting or questioning by the teacher.

Based on the results of Conradys (2015) study, it can be concluded that explicit

instruction alone is not enough to fully develop all components of metacognitive thinking.

Instead, several studies suggest that explicit instruction be strategically taught using various

contexts, examples, and applications along with strategies for developing metacognitive

regulation skills such as comprehension monitoring, metacognitive training or metacognitive

scaffolding (Schurter, 2002; Kramarski Mevarech, 2003; Jbeili, 2012). In a research study,

Schurter investigated the effects of using comprehension monitoring, problem solving strategies,

and explicit instruction on student problem solving abilities in three university level remedial

mathematics courses totaling 60 students. Comprehension monitoring is a metacognitive

regulation technique that provides students self-questioning techniques that guide them through

problem solving. Findings showed that students who were taught comprehension monitoring

alone or in conjunction with problem solving strategies outperformed those that were taught with

traditional instruction. However, the results indicated that there was no significant difference

between the two treatment groups. In conclusion, the findings of this study support the position
 15

that the implementation of comprehension monitoring strategies alone or accompanied by

problem solving strategies can improve performance in mathematical problem solving.

In a similar study, Kramarski and Meravech (2003) investigated the effects of a

metacognitive regulation technique called metacognitive training on mathematical reasoning 12

eighth grade classrooms totaling 384 students. Metacognitive training is a metacognitive

regulation strategy like comprehension monitoring. Using metacognitive training, students were

taught how to use self-questioning techniques during problem solving. Findings from this study

echoed Schurters findings, suggesting that metacognitive training directly contributed to the

improvement of mathematical reasoning. Moreover, the group that was exposed to cooperative

learning and metacognitive strategies (COOP+META) outperformed their counterparts, the

independent and metacognitive group (IND+META), which in turn outperformed the

cooperative group (COOP) and the independent group (IND). No significant difference was

found between the two groups, COOP and IND, that were not exposed to metacognitive

strategies. In conclusion, evidence from this study showed that the two treatment groups,

COOP+META and IND+META, developed fluency and flexible thinking, better knowledge

transfer, and the ability to utilize logical-formal arguments as a direct result of using

metacognitive regulation strategies.

In a similar study to Kramarski and Meravech (2003), Jbeili (2012) examined the effect

of metacognitive scaffolding embedded within cooperative learning on 240 fifth grade students

mathematics conceptual understanding and procedural fluency in problem solving. The result of

Jbeilis study coincided with Schurter (2002) and Kramarski and Mevarechs (2003) findings

that confirmed the importance of learning strategies that used scaffolding techniques to develop

metacognitive regulation skills. The data indicated that the group taught using cooperative
 16

learning with metacognitive scaffolding (CLMS) significantly outperformed their counterparts,

the cooperative learning group (CL) which in turn outperformed their counterparts, the

traditional group (T). Jbeili concluded that the CLMS group outperformed both its counterparts

in problem solving that required conceptual understanding and procedural fluency because they

were provided with various strategies to support this outcome. To elaborate, the CLMS group

surpassed their counterparts because they were able to work cooperatively using metacognitive

questions. The metacognitive questioning helped prompt students to construct their knowledge

and skills by assisting them in retrieving prior knowledge as well as connecting it to new

knowledge, building and reinforcing schema. It enabled them to evaluate problems and connect

it to similar past problems improving accuracy and efficiency in problem solving. It provided

them with flexible thinking and multiple approaches through group discussion and support of the

metacognitive questions. It facilitated cooperation and deep learning by requiring them to

explain their thinking to group members during discussion rather than relying on rote memory.

It can be concluded that all of the skills described above assisted the CLMS group in easily

remembering and retrieving math concepts and problem solving strategies and thus, the reason

for their high achievement.

Reflective Journal Writing. U.S. teachers often communicate that their students

struggle with problem solving. During problem solving, instructors observed that students spent

very little time planning, quickly chose one strategy to apply and never reflected on the

effectiveness of the strategy as they chugged away and eventually gave up. Olson & Johnson

(2012) investigated the value of journal writing in mathematics for two groups of eighth grade

students totaling 107 students. Their findings indicated that the treatment group showed greater

achievement than the group who did not engage in journal writing. Furthermore, the study also
 17

indicated that recording thinking steps during problem solving had several benefits. Olson and

Johnson (2012) reported that journal writing allowed students to reflect, monitor, and evaluate

their mathematical problem solving, it promoted critical thinking skills such as analyzing data,

evaluating and comparing facts, and synthesizing information, and it allowed instructors to

assess students mathematical thinking and provided regular feedback to deepen the

understanding of a concept or correct student thinking. In conclusion, the results of this study

showed that regular use of journal writing improved both student academic achievement as well

as attitude towards mathematics.

Socratic Questioning and Seminars. In Conradys (2015) study, students frequently

struggled to work through an idea on their own relying heavily on teacher questioning or

prompting to develop their remaining thought. Theorists posit that learning is constructed in

exactly this wayquestioning. Vygotskys (1986) theory of cognitive development describes

learning as a social process. Likewise, the mathematics learning process is also described as an

inherently social activity in which students develop metacognitive thinking skills through

sharing, comparing, and evaluating mathematical strategies among their peers in order to

determine the best approach (Conrady, 2015; Pintrich, 2002; Schoenfeld, 1992). Researchers

hypothesized that cooperative learning accompanied by reflective discussion helped students

develop critical thinking skills and deepen their understanding of concepts and procedures

(Jbeili, 2012; Kramarski & Mevarachi, 2003; Schurter, 2015).

Several studies suggest that Socratic Questioning and Socratic Seminars are effective

strategies for developing these skills. In a study, Tanner and Casados (1998) examined the

contributions of Socratic Seminars in a Trigonometry, Statistics, and Functions class of 17 high

school students. Data was collected and analyzed using videotaping, surveys, and research
 18

journal notes. The findings of the study concluded that using Socratic Seminars allowed students

to talk through ideas and as a result, students became insightful, logical mathematical solvers.

Tanner and Casados findings coincided with Olson and Johnsons (2012) findings, illustrating

that student mathematics disposition, active participation, and articulation of concepts all

increased as a result of Socratic Questioning and Seminars. Yang (2008) conducted a similar

study investigating the effects of Socratic Questioning or dialoging in online discussion on

critical thinking skills. The results of her study corroborated previous studies. The data

suggested that the use and modeling of Socratic Questioning had a positive effect on student

critical thinking skills. It aided students in constructing new meaning from content, enabled

them to think independently and critically, explore applications to problems, and generate

thoughtful questions (Tanner & Casados, 1998; Yang, 2008).

Methodology

Research Question

The purpose of this study was to determine the effectiveness of developing critical

thinking skills on student standardized test performance. The study explored the following

questions: How does the development of metacognitive knowledge and regulation skills support

conceptual understanding and procedural fluency? What are the effects of keeping a

metacognitive journal on the development of critical thinking? What role does feedback and

constructivism play in developing critical thinking skills? How does consistent feedback affect

the development of critical thinking skills?

Research Design

The study incorporated a mixed methods approach, integrating a quasi-experimental

design with a naturalistic study. The variable in the study design was critical thinking skills in
 19

the form of metacognitive thinking skills of knowledge and regulation.

Participants

The participants of this study included 60 eighth grade students, 29 females and 31 males,

selected from a middle school in California. The demographics of the participants consisted of

students with diverse ethnic backgrounds (African American 8.3%, Hispanic 36.4%, Asian 7.1%,

Filipino 25.9%, White 13.6%, Native American 0.4%, Pacific Islander 0.9% and Two or More

Races 6.6%). Socioeconomically disadvantaged students accounted for 43.7% of the participants

and English Learners (ELs) accounted for 8.3% of the participants. Students with disabilities

were not included. The treatment group (T) and control group (C) both consisted of 30 students

each. The treatment group was chosen randomly between two classes with similar demographics

and ability levels. The treatment group was comprised of 16 females and 14 males while the

control group had 13 females and 17 males.

Conditions

The study was conducted in the normal classroom setting over a period of two weeks.

Both the treatment and control group were taught by the same teacher. Classes followed a block

schedule lasting 84 minutes every day except for Wednesday which was only 64 minutes long.

The content covered during the treatment was taught previously at the beginning of the school

year, more than five months prior.

Measurements

In the study, three measures were used to assess students mathematical achievement,

fluency, and metacognitive knowledge: the California Assessment of Student Performance and

Progress (CAASPP) practice test, a Likert scale self-assessment, and a metacognitive journal.

To analyze the quantitative data, a combination of descriptive and inferential statistics was used.
 20

The data analysis included two types of inferential statistic tests, an independent samples t-test

and a paired samples t-test. The independent samples t-test was used to compare the mean

difference between the treatment and control group assessment scores while the paired samples t-

test was used to compare the mean difference between the pre- and post-test scores within the

treatment group. The constant comparative method was used to analyze the qualitative data.

Coding was used to identify the evidence of metacognitive thinking and types of thinking during

problem solving.

Pre-and Post-Test. The first measure was a pre- and post-test. In order to produce valid

and reliable data, the pre- and post-test were identical, consisting of eight carefully selected

questions from the official CAASPP practice test as the unbiased measurement tool to simulate a

standardized test. The test questions were specifically chosen to represent a wide range of

concepts and procedures with varying difficulty requiring lower and higher order thinking skills:

understanding, identifying, applying, interpreting, evaluating, and problem solving. The

concepts and procedures assessed within these few problems were the ability to understand,

analyze, identify and interpret information from graphs and word problems, construct and solve

an equation from a graph and a word problem, and write and solve systems of linear equations in

two ways, graphically and algebraically.

The test was mixed with multiple choice and short answer questions which assessed

students content knowledge of linear functions and equations. The pre- and post-test were

administered before and after the treatment and scored out of nine points according to the

CAASPP rubric. One question had a value of two points because it required two answers, hence

nine points for eight questions. The pre-test was used to establish students current achievement

and to assess whether the abilities of the treatment and control group were similar before the
 21

treatment began. The post-test was used to determine whether the treatment had a statistically

significant effect on students standardized test scores within the treatment group as well as

between the treatment and control group. First, an independent t-test was used to compare the

pre- and post-test scores to determine whether there was a statistically significant difference

between the means of the treatment and control group before and after the treatment. Second, a

paired samples t-test was used to determine if there was a statistically significant difference

between the pre- and post-test scores within the treatment group.

Self-Assessment. The second measure was a Likert scale self-assessment which assessed

students disposition and perception towards their test performance, ability level, and confidence

in their concept mastery before the treatment and after. In the four question self-assessment,

students were asked to use a rating scale of 1-5. The questions asked students how well they

think they scored, how well they remembered the test topic, how easy they felt the questions

were, and how well they felt they could take a test without the aid of notes.

Metacognitive Journal. The third measure consisted of two types of metacognitive

journals, the first was adapted from a double entry journal with embedded metacognitive

scaffolding (MS) questions to help students plan, monitor, and evaluate their problem solving

process and the second was a reflective writing piece. In essence, the MS journal was a graphic

organizer and self-questioning checklist in one. The MS journal was divided into three sections:

planning phase, monitoring, phase, and evaluating phase. The planning phase included

metacognitive guiding questions to activate and connect students prior knowledge to new

information. The monitoring phase required students to document the implementation of their

problem solving strategy detailing steps in the first column and explanations and justifications of

those steps in the second. In the third column, the metacognitive guiding questions were listed to
 22

help students monitor the effectiveness of their strategy. The evaluation phase asked the students

to think about their final answer and determine whether it made sense, providing evidence and

reasoning to support their claim. This section also contained guiding questions to help students

reflect on their process, evaluate the effectiveness of their strategy and determine how it could be

applied to future problems or other contexts to make the process more efficient. The reflective

journal asked students to respond to three prompts: (a) Explain the importance of tables, graphs,

and equations in problem solving and how it transfer to other contexts and real life (b) Explain

the different approaches that could be used to write a linear equation, and (c) Compare two

different options making a claim and supporting it with mathematical evidence and reasoning.

The journals provided qualitative data on the use and types of metacognitive thinking used

during mathematical problem solving of complex non-routine problems.

Procedure

Treatment Group. Before the treatment, students completed the pre-test and self-

assessment. After students completed the pre-test and self-assessment, they were asked to

complete a Metacognitive Awareness Inventory (MAI), form groups of three to read an article

about metacognition, and take Cornell notes. Next, the instructor conducted a whole class

discussion on metacognition, prompting the students to share information on what they learned

after completing the MAI and reading the article. The instructor provided feedback throughout

the discussion, usually through questioning and prompting to help students develop or deepen

their thinking. After the discussion, the students responded to a reflective journal prompt on

their thoughts and questions about metacognition, explaining what they learned about themselves

as metacognitive thinkers, and what metacognitive skill they would like to improve on and why.

During the rest of the two week study, the instructor used explicit instruction to model
 23

metacognitive thinking using various examples, contexts, and application. Students were given

simple scenarios and asked to practice modeling the metacognitive thinking process they had just

learned, working within their group. After students shared their thinking process with the whole

class, the instructor modeled her own metacognitive thinking or expert behavior using a think

aloud method. This exercise was done regularly usually at the beginning of each class meeting

for a period of two weeks. The problems and scenarios gradually became more complex.

During the study, students completed two performance tasks (PT) that required them to

use critical thinking skills in order to successfully problem solve. The two PT were non-routine

real-world problems. The first PT was called Using Water Wisely, a seventh grade level

problem, obtained from the official CAASPP website. Students were given the opportunity to

first work within their triads to solve the multi-part PT. Learning was scaffolded with the use of

a metacognitive journal. Students completed the problem with the use of the MS questions,

documenting their steps and their thinking during the planning, monitoring, and evaluating

phase. When students struggled within their groups, they requested teacher feedback. The

teacher used prompting and questioning to help them think through their strategies. Once the

students finished, the instructor asked different groups to share their answers and strategies. The

teacher used Socratic Questioning techniques to guide and develop student thinking when

necessary. After students shared their answers, the teacher used a think aloud once again to

model her thinking process for students to compare with their own.

Before the second PT, students were given an article about health insurance in order to

gain some background knowledge and learn new vocabulary on the unfamiliar topic. They first

worked in their groups to take cornell notes then engaged in a whole class discussion about

health insurance to clarify or dispel confusion before starting the actual PT. The second PT was
 24

designed by the instructor, titled Choosing a Health Insurance Plan. It was also a multi-part

problem, much more cognitively complex than the previous PT. The PT required students to

analyze information and to model the situation with various representations: tables, graphs, and

equations. It required students to use critical thinking skills to construct linear

functions/equations and use it to draw conclusions or make predictions about a real world

problem as well as provide reasoning and evidence to back up claims. The students were given

the metacognitive journal and five days to work on the five problems and its subcomponents. On

the ninth day, in their triads, the students compiled their metacognitive journals and prepared

questions for their Socratic Seminar the next day. Students facilitated the discussion, generating

questions and comments about health insurance plans using Socratic Seminar sentence starters.

When students got stuck, the instructor would provide prompting or new questions to think

about. Several students were able to successfully model their questions after the instructors

helping to facilitate group discussion and smooth transitions. Students debriefed about the

process and completed a final metacognitive journal reflection. Students took the post-test and

self-assessment the next day.

Control Group. The control group was instructed in a traditional manner. They took the

pre- and post-test as well as the self-assessment. The instructor modeled problem solving steps

using a think aloud but did not discuss any metacognitive thinking strategies or use questioning

techniques. The teacher demonstrated problem solving steps and clarified any lingering

questions. Like the treatment group, the control group worked cooperatively to complete the two

PTs. Instead of a Socratic Seminar, students presented a problem from the PT.

Results

Quantitative Data Analysis

 25

Pre- and Post-Test. In order to determine the homogeneity of the two groups before the

study, an initial independent samples t-test was conducted on the pre-test scores (Table 1).

Table 1
Comparing Treatment and Control Group Pre-Test Scores for Homogeneity
Pre-Test

Source of Variation N M SD SEM t p-value

Treatment 25 2.6 1.71 0.34 0.88 0.3829

Control 27 3 1.57 0.30

*p < .05

Results showed that the pre-test scores of the treatment and control group showed no

statistically significant difference between the means of the two groups, t (50) = 0.88; p = 0.3829

with significance at p < .05. Thus, the ability level of the two groups was similar before the

beginning of the treatment.

Inferential Statistics. The data from the pre- and post-test scores of the treatment group

were analyzed using a paired samples t-test (Table 2 & 3) to determine statistical significance.

Table 2
Paired Samples Statistics of Treatment Group Pre- and Post-Test Scores
N M SD SEM

Pre-Test 25 2.6 1.71 0.34

Post-Test 25 5.44 1.83 0.37

Table 3
Paired Samples t-test on Treatment Group Pre- and Post-Test Scores
95% CI of the
M SD SEM Mean t df p
Difference
Pre-Test Scores
- -2.84 1.31 0.26 -3.38, -2.30 10.82 24 *p < 0.0001
Post-Test Scores
*p < .05
 26

The results of the paired samples t-test showed an extremely statistically significant

difference between the pre-test (M = 2.6, SD = 1.71) and post-test scores (M = 5.44, SD = 1.83); t

(24) = 10.82, p < 0.0001, confirming > 99% confidence level that the treatment contributed to

the increase in students scores.

In order to compare the post-test scores between the treatment and control group, results

were analyzed using an independent samples t-test (Table 4).

Table 4
Comparing Treatment and Control Group Post-Test Scores for Statistical Significance
Post-Test

Source of Variation N M SD SEM t df p-value

Treatment 28 5.46 1.75 0.33 3.34 54 *0.0015

Control 28 3.82 1.93 0.36

*p < .05
The analysis revealed an extremely statistically significant difference between the two

groups, t (28) = 3.34; p = 0.0015, significant at p < .05, confirming > 99% confidence level that

the treatment contributed to the increase in students scores.

Descriptive Statistics. The bar graphs in Figure 1 represent the pre- and post-test scores

for the treatment and control group. The mean, median, mode, and standard deviation were used

to compare the test performance between the two groups.

 27

Figure 1. Treatment and Control Group Pre-Test and Post-Test Scores

The central tendency and measures of variability on the pre-test scores for both the

treatment group (M = 2.6, SD = 1.71, Md = 3, Mo = 3) and control group (M = 3, SD = 1.57, Md

= 3, Mo = 3, 4) were similar. However, based on the post-test data results, the treatment group

(M = 5.44, SD = 1.75, Md = 5, Mo = 5) significantly outperformed the control group (M = 3.82,

SD = 1.93, Md = 4, Mo = 3, 5). The mean post-test scores from the treatment group increased

2.84 points (109%) while the control only increased 0.82 points (27%). The data also indicated

that the pre-test scores of both groups had similarly non-symmetrical right skewed distributions.

Scores were clustered to the left suggesting that the majority of the participants were performing

below proficiency. However, the data from the post-test suggests that the treatment group

became more left skewed, with most scores clustering in the higher range, right part of the graph

showing huge improvement. The control groups scores remained slightly right skewed

indicating that many of the students in that group were still performing below proficiency in

addition to some improvement overall.

Self-Assessment. Descriptive statistics (Table 5) was used to analyze the data from the

self-assessment to establish students general disposition towards the test before and after. The

first question asked students to rate how well they thought they scored on the test. The mean of

the treatment group increased 18% while the control group increased 2%. The second question
 28

asked students to rate how well they felt they remembered the test topic. The data showed that

the treatment group increased 9% while the control group increased 11%. The third question

asked students to rate the ease of the test questions from 1-5, 5 being very hard. The results

indicated that the mean decreased 3% for the treatment and 9% for the control. The last question

asked students to rate how well they think they would do on tests without the aid of notes. In the

treatment group, scores decreased 11% while scores increased 12% in the control group.

Table 5
Descriptive Statistics of Question 1 on Self-Assessment

Question Treatment Control Question Treatment Control

1 2

M Md Mo M Md Mo M Md Mo M Md Mo

Pre 2.54 3 3 2.38 2.5 3 2.67 3 3 2.31 2 2

Post 3 3 3 2.43 2 2 2.91 3 3 2.57 3 3

Question Treatment Control Question Treatment Control

3 4

M Md Mo M Md Mo M Md Mo M Md Mo

Pre 2.92 3 3 3.23 3 3 3.29 3 3 2.19 2 2

Post 2.83 3 3 2.93 3 3 2.91 3 3 2.46 3 3

Qualitative Data

MS Journals. Two types of metacognitive journals were assessed. In the MS journal, a

constant comparative method was used to analyze the data for frequency and types of

metacognitive thinking. 26 journals were collected out of 30 students. The data was analyzed
 29

and coded in four categories: (a) use of declarative knowledge, (b) use of declarative and

procedural knowledge, (c) use of declarative, procedural, and conditional knowledge, and (d) use

of declarative, procedural, conditional, and regulatory knowledge. 23.1% demonstrated use of

declarative and procedural knowledge, 53.8% demonstrated use of declarative, procedural, and

conditional knowledge, and 23.1% demonstrated the use of all four metacognitive thinking skills.

Those who showed all four thinking skills were able to show all mathematical problem solving

steps, complete the explanation and justification column explaining what they were doing, how

they were doing it, why they were doing it, and they kept track of their process by jotting down

questions in the third column of the journal about what they were confused about to remind

themselves to ask the teacher or a peer. One student wrote down, How am I going to graph it?

Another wrote down, How do you find the total cost of the platinum HMO? and What are

some strategies for finding the equation that models a given situation? The data showed that

overall, 76.9% of the students were utilizing procedural and conditional knowledge. They knew

how to apply math concepts or skills and knew when and why to apply them.

Reflective Journals. The reflective writing task was analyzed to assess the rate in

which students demonstrated critical thinking skills of three types: (a) explain importance of

knowledge and skills in the PT and other real life contexts, (b) describe different approaches, and

(c) choose the best option for a given situation and justify claims with evidence and reasoning.

23 journal reflections were collected and analyzed. For the first question, 95.7% could explain

importance of tables, graphs, and equations in the context of the PT but only 72.7% of those

students connected their knowledge to the real world. Some students articulated that tables,

graphs, and equations modeled situations, allowing them to compare, contrast, and predict

different costs to make the best decision. One student wrote, The importance of making tables,
 30

graphs, and equations are tables can help seeing how something increases and decreases over

time, graphs can help figure out if two [lines] intersect, which means at one point, theyre the

same, and equations help with figuring out if something is more expensive or cheap. Also, both

the graph and equation can help find the breakeven point. Another student wrote, A graph lets

you see if any points intersect, if the points intersect then that represents if the two lines ever

costs the same money, or if they ever spend the same time, etc. With these techniques you can

organize the different data in order to have a clearer representation for contrast. One more

student wrote, It is important because if we didnt have any equation how would [we] be able to

solve the problems? Plus if we didnt have a table we wouldnt be able to make the graph and

that graph could be helpful to check how much they charge a month for bills. If you didnt check

you could of got ripped off. Equations can help us find the slope or where they break even or

they intersect. For the second question, 56.5% identified only one strategy to find the

components of a linear equation and 43.5% identified at least two strategies. For the last

question, 95.7% made a claim as to which health insurance plan on the PT was the best and

provided mathematical evidence and reasoning to support that claim. Only one student failed to

do so.

Discussion

The goal of this study was to determine whether developing critical thinking skills would

likely lead to improved standardized test scores. More specifically, the study investigated

whether teaching metacognitive thinking skills would lead to higher mathematical achievement

on standardized assessments and lead to better conceptual understanding and procedural fluency.

The data showed that every students test score improved except one which stayed the same.

The mean increased from 2.6 to 5.44, showing a 109% gain. The results of the statistical analysis
 31

on the pre-test and post-test scores indicated an extremely statistically significant difference

between the two means at > 99% confidence level. The results suggested that the improvement

in test scores was directly influenced by the treatment. Thus, the outcome of this study supports

the hypothesis that metacognitive strategies lead to higher mathematical achievement on

standardized test performance. This implies that teaching students metacognitive strategies is

vital to developing students critical thinking and mathematical problem solving.

Implications for Instruction

Based on the results, implications for teaching include implementing all or part of a

broad repertoire of instructional strategies for developing metacognitive thinking skills: explicit

instruction, metacognitive scaffolding, Socratic Questioning, and Socratic Seminars. Findings in

this study directly coincide with the findings of other studies regarding the benefits of

metacognition in mathematical problem solving. Metacognitive skills should be explicitly taught

using a wide range of examples, contexts, and application for the best transfer of knowledge.

Effective strategies for accomplishing this include using a think aloud to model the thinking

process. Moreover, opportunities should be provided to students to practice these skills with

increasing complexity as to not overload them cognitively. Metacognitive scaffolding and

Socratic Questioning should be used in conjunction with journal writing to develop students

planning, monitoring, and evaluating of their thinking process. Metacognitive scaffolding allows

students to develop self-questioning techniques so that they can connect prior knowledge to new

information allowing students to become more autonomous. Socratic Questioning is an effective

technique that can be easily implemented in the classroom to guide students to think logically

and achieve deeper understanding. It also provides valuable feedback for students and teachers.

Last, Socratic Seminars are an effective method for students to share thinking with peers so they
 32

can compare strategies with one another and strengthen their mathematical fluency.

Limitations

Limitations of this study are that the number of participants is small and that it is

conducted in a very short period of time, over the course of two weeks, therefore various factors

could impact the results such as absences. Absences were a big factor because it means loss of

time to develop and practice skills, to engage in peer discussion, and receive teacher feedback.

Moreover, there were several events that impacted the student's mental state or ability to

concentrate. During the two weeks of treatment, there was a lot of stress because of state testing

and learning time was loss because of school site activities (8th grade panoramic pictures, fire

drill, 8th grade field trip and promotion discussion, music festival field trip, etc.). There were

also limitations because the quantitative and qualitative measures of the study were not designed

to measure conceptual understanding and procedural fluency, it only measured achievement.

Conclusion

The 21st century has raised the bar quite high for achieving success, requiring those who

enter adulthood and the professional scene to possess specialized critical thinking skills. These

are the skills necessary to enter a highly competitive workforce. With the shift to technology and

advancements, STEM has become one of the fastest growing careers. The implication for

teachers is that it has become even more crucial to prepare students for these STEM jobs.

Hence, it is important to understand how to most effectively develop students critical thinking

skills, specifically fluency and flexibility of thinking. This study explored critical thinking and

metacognition, highlighting literature on teaching strategies that develop these skills. The results

of the study concluded that teaching metacognitive strategies had a positive impact on

mathematical achievement.
 33

Recommendations for Future Study

There have been abundant studies on critical thinking and metacognition but there is still

so much more that needs to be explored. For example, the results of this study corroborated the

findings of other research to some extent. Student achievement did increase as a result of the

treatment, but several metacognitive strategies were employed as a whole so it is unclear as to

which specific strategies were most effective and whether the strategies can be used

independently. Furthermore, one of the goals of this study was to find out whether the

development of metacognitive thinking skills improved conceptual understanding (CU) and

procedural fluency (PF). CU and PF require mastery of concepts and skills in order to easily

remember, retrieve, and correctly apply the information. The National Council of Teachers of

Mathematics (NCTM) define PF as a critical component of mathematical proficiency. PF is

described as the ability to apply procedurals accurately, efficiently, and flexibly; to transfer

procedures to different problems and contexts; to build or modify procedures from other

procedures; to recognize when on strategy or procedure is more appropriate to apply than

another. The results of this study could not verify this outcome. It is not clear as to whether the

increase in test scores demonstrates that there was an increase in CU and PF. There is a chance

that the increased test scores were due to knowledge obtained through rote memorization.

Therefore, further study is needed to determine effects of metacognition on CU and PF as

well as explore which specific strategies are most effective for developing and assessing CU and

PF. Moreover, several studies in the literature review mentioned that teaching metacognition

supported cooperative learning and as a result, improved students mathematical dispositions as

well as the classroom environment. The self-assessment showed increases in some areas for

both groups, such as students perception of their test performance after the test as well as
 34

remembering the test topic. Perception of difficulty level decreased so students did find the test

questions to be easier on the post-test. However, there was a surprising decrease in student

confidence on their ability to test without the aid of some type of notes for the treatment group

but not the control group. The increases and decreases were not statistically significant but this

too needs more data in order to explore the applications of metacognition on classroom

environment and disposition.

Furthermore, as a result of analyzing the metacognitive journals and observing the

Socratic Seminar, questions arose about how metacognition might improve the articulation of

mathematics, both written and oral which is a 21st century skill and desirable skill according to

businesses and non-profit leaders. There is no dispute that the benefits of critical thinking and

metacognition are well documented but further studies in secondary level mathematics is still

much needed. In conclusion, more in depth studies are needed to determine how teachers can

best support all students, especially the most vulnerable populations, in developing critical

thinking skills as a way to address inequities and close the achievement gap.
 35

References

California Department of Education. (2016). California Assessment of Student Performance and

Progress (CAASPP) Results. Retrieved from http://www.caaspp.cde.ca.gov

Conrady, K. (2015). Modeling metacognition: Making thinking visible in a content course for

teachers. REDIMAT, 4(2), 132-160. doi: 10.4471/redimat.2015.1422

Darling-Hammond, L. (2009). The flat world and education: How Americas commitment to

equity will determine our future. New York: Teachers College Press.

Dewey, J. (1910). How we think. Boston: D.C. Heath & Co. [Kindle DX version]. Retrieved

from Amazon.com

Ennis, R. H. (1985). A logical basis for measuring critical thinking skills. Educational

Leadership, 43(2), 44-48.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-

developmental inquiry. American Psychologist, 34(10), 906-911. doi: 10.1037/0003-

066X.34.10.906

Garofalo, J., & Lester Jr, F. K. (1985). Metacognition, cognitive monitoring, and mathematical

performance. Journal for research in mathematics education, 163-176. Retrieved from

http://www.jstor.corg/stable/748391

Goos, M., Galbraith, P., & Renshaw, P. (2002). Socially mediated metacognition: Creating

collaborative zones of proximal development in small group problem solving.

Educational Studies in Mathematics, 49(2), 193-223. doi: 10.1023/A:1016209010120

Gray, S. S. (1991). Ideas in Practice: Metacognition and Mathematical Problem Solving. Journal

of developmental Education, 14(3), 24-26, 28. Retrieved from

http://www.jstor.org/stable/42775523
 36

Jbeili, I. (2012). The effect of cooperative learning with metacognitive scaffolding on

mathematics conceptual understanding and procedural fluency. International Journal for

Research in Education (IJRE), 32, 45-71.

Kramarski, B. & Mevarech, Z. R. (2003). Enhancing mathematical reasoning in the classroom:

The effects of cooperative learning and metacognitive training. American Educational

Research Journal, 40(1), 281-310.

Kurfiss, J. G. (1988). Critical thinking: Theory, research, practice, and possibilities (Report No.

0-913317-44-6). Washington, DC: Association for the Study of Higher Education.

Lin, X. (2001). Designing metacognitive activities. Educational Technology Research and

Development, 49(2), 23-40. doi: 10.1007/BF02504926

Mayer, R. E. (1993). Understanding individual differences in mathematical problem solving:

Towards a research agenda. Learning Disability Quarterly. 16(1), 2-5.

National Center for Education Statistics (2013). The nation's report card: A first look: 2013

mathematics and reading (NCES 2014451). National Center for Education Statistics,

Institute of Education Sciences, U.S. Department of Education, Washington, D.C.

National Center for Education Statistics (2015). Highlights from TIMSS and TIMSS advanced

2015. (NCES 201702). National Center for Education Statistics, Institute of Education

Sciences, U.S. Department of Education, Washington D.C. Retrieved from

https://nces.ed.gov/pubs2017/2017002.pdf

National Council of Teachers of Mathematics (2014). Procedural fluency in mathematics a

position of the National Council of Teachers of Mathematics. Retrieved from

http://www.nctm.org/Standards-and-Positions/Position-Statements/Procedural-Fluency-

in-Mathematics/
 37

OBrien, S. (2015, June 15). Math, science skills add up to more job opportunities: Survey.

CNBC LLC. Retrieved from http://www.cnbc.com/2015/06/15/math-science-skills-add-

up-to-more-job-opportunities-survey.html

Olson, J. & Johnson, C. (2012). Implementing journal writing in a grade 8 mathematics class.

Academic Research International, 3(3), 191-197.

Pammu, A., Amir, Z., & Maasum, Tg. (2011). Metacognition in reading: Reviewing the

literature. SoLLs.INTEC 2011 Proceedings. Retrieved from

http://slim.ukm.My/solls/SOLLS_Proceeding_2011.pdf

Paul, R. & Elder, L. (2006). Critical thinking: Tools for taking charge of your learning and your

life (2nd ed.). Upper Saddle River, NJ: Pearson Prentice Hall.

Paul, R. & Elder, L. (2014). The thinkers guide to the art of Socratic Questioning [Kindle DX

version]. Retrieved from Amazon.com

Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and

assessment. Theory into Practice, 41(4), 213-225.

Schoenfeld, A. H. (1992). Learning to think mathematically: Problem solving, metacognition,

and sense making in mathematics. Handbook of research on mathematics teaching and

learning, 334-370.

Schraw, G. & Moshman, D. (1995). Metacognitive theories. Education Psychology Review, 7

(14), 351-371.

Schurter, W. A. (2002). Comprehension monitoring: An aid to mathematical problem solving.

Journal of Developmental Education, 26(2), 22-4, 26, 30, 32-33.

Su, H.F., Ricci, F.A., & Mnatsakanian, M. (2016). Mathematical teaching strategies: Pathways to

critical thinking and metacognition. Journal of Research in Education and Science

 38

(IJRES), 2 (1), 190-200.

Swanson, H. L. (1990). Influence of metacognitive knowledge and aptitude on problem solving.

Journal of Educational Psychology, 82(2), 306-314

Tanner, M. L. & Casados, L. (1998). Promoting and studying discussions in math classes.

Journal of Adolescent & Adult Literacy, 41(5), 342-350. Retrieved from

http://www.jstor.org/stable/40016937

Vygotsky, L. S. (2012). Thought and language. Cambridge, MA: MIT Press.

Yang, Y. C. (2008). A catalyst for teaching critical thinking in a large university class in Taiwan:

Asynchronous online discussion with the facilitation of teaching assistances. Educational

Technology Research and Development, 56(3), 241-264.