Research Report
Medical Student Progress Examination Performance
and Its Relationship With Metacognition, Critical
Thinking, and Self-Regulated Learning Strategies
Chi Chang, PhD, MS, Migdalisel Colón-Berlingeri, PhD, Brian Mavis, PhD,
Heather S. Laird-Fick, MD, MPH, Carol Parker, PhD, MPH, and David Solomon, PhD
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
Abstract
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/29/2023
Purpose
Metacognition and critical thinking
are essential for academic success. The
relationship between these components
and medical student learning, as
assessed with progress examinations,
informs curriculum development and
efforts to ensure learning progression of
all students. This study assessed learning
mechanisms by modeling medical
students’ progress test performance
longitudinally at Michigan State
University College of Human Medicine.
Method
Medical students’ (n = 184) medical
knowledge was assessed 5 times
from fall 2017 through spring 2019
using the Comprehensive Basic
Science Examination (CBSE). Structural
equation modeling was conducted to
investigate associations between 3 latent
Metacognition, or knowledge
concerning one’s own cognitive processes,
has been found to be an important
component of student academic success.1
It is one of the cognitive strategies
developed by the lifelong learner.2
Commitment to lifelong learning is in
Please see the end of this article for information
about the authors.
The authors have informed the journal that they
agree that both Chi Chang and Migdalisel Colón-
Berlingeri completed the intellectual and other work
typical of the first author.
Correspondence should be addressed to Chi
Chang, Office of Medical Education Research and
Development, and Department of Epidemiology
and Biostatistics, College of Human Medicine,
Michigan State University, 965 Wilson Rd., A214C,
East Lansing, MI 48824; telephone: (517) 353-3454;
email: chisq@msu.edu.
Acad Med. 2021;96:278–284.
First published online September 29, 2020
doi: 10.1097/ACM.0000000000003766
Copyright © 2020 by the Association of American
Medical Colleges
Supplemental digital content for this article is
available at http://links.lww.com/ACADMED/B29.
278
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
structures—metacognitive awareness,
critical thinking, and self-regulation—and
their relationship with students’ initial
CBSE scores and growth in such scores.
The authors measured metacognitive
knowledge and regulation by the
Metacognitive Awareness Inventory,
critical thinking skills by the Watson-
Glaser Critical Thinking Appraisal, and
self-regulation by the Learning and Study
Strategies Inventory.
Results
Students’ aggregate performance on
5 CBSE scores grew 31.0% the first
semester, 16.5% the second semester,
30.1% the third semester, and 22.4%
the last semester. Critical thinking had
a significant positive relationship with
initial performance (β = 1.956, P < .001),
self-regulation had a significant positive
relationship with growth (β = 3.287,
turn a significant component of the
medical profession.3 Expert adult learners
use metacognitive skills to plan and select
learning strategies; monitor and evaluate
the effectiveness of learning strategies
through self-assessment and review;
adjust learning behaviors, processes,
and strategies; and reflect on learning.
Teaching explicitly about metacognition
and asking students to use their
metacognitive skills has been found to
enhance learning.4 Schraw and Dennison
define metacognition as the ability to
reflect on, understand, and control
learning.5 According to Zimmerman,6
metacognition is part of the process
of self-regulated learning and includes
planning, goal setting, organization,
self-monitoring, and self-evaluation at
various points during the process of
knowledge acquisition.
The theory of metacognitive knowledge
and awareness in psychological and
educational research cuts across different
theoretical approaches within learning
Academic Medicine, Vol. 96, No. 2 / February 2021
P < .05), and metacognitive awareness
had a negative relationship with growth
of student performance in the progress
test (β = −3.426, P < .01).
Conclusions
This structural equation framework is
useful for examining the relationships
among 3 latent structures—critical
thinking, metacognition, and self-
regulation—and their relationships
with students’ progress scores in
academic achievement. The initial status
of progress examination scores was
explained by students’ critical thinking
ability, but their learning growth on the
progress scores was explained by their
self-regulation and metacognitive ability.
These findings help explain student
performance on standardized progress
examinations and can aid in interventions
to promote student success.
and development research7: neo-Piagetian
models8 of cognitive development,
information processing models9 in
cognitive science, and situated learning
models10 in social cognition process.
Situated learning emphasizes that
knowledge is constructed through the
interactive processes between a learner’s
mind, behavior, and environment.
The 3 constructs—critical thinking,
metacognition awareness, and self-
regulation—can be assessed, connected,
and explored by this interaction.11 These
theoretical perspectives provide the
framework for current curriculum reform
in most medical schools, as students,
especially professional students, tend to
learn better when they are more aware of
their thinking and cognition.12
Similar to critical thinking, metacognition
awareness, and self-regulation, learning
tends to progress better for students if
they have steady improvement during
their learning process. However,
quantifying the learning developmental

process or performance growth is
not easy. One main reason is that
measuring the growth of domain
knowledge requires a standardized
testing procedure and comparable testing
items for the domain each time the
test is administered. The same domain
is repeatedly measured by items with
similar psychometric properties (such as
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
same item difficulties), so that the scores
are comparable across time points and
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/29/2023
the change in scores across time points
can be regarded as growth in cognitive
knowledge.
Progress tests are the assessment tools
designed for evaluating development
of cognitive knowledge, specifically
during a learning process that fulfilled
requirements for measuring the growth
in academic performance, which was the
focus of this study. Medical schools will
gradually rely on assessment tools for
diagnosing student domain knowledge
acquisition longitudinally, when the
implementation of integrated curriculum
and flipped-classroom instructional
strategy becomes popular.
At Michigan State University’s (MSU’s)
College of Human Medicine (CHM),
the shared-discovery curriculum
(SDC) includes implementation of
flipped-classroom learning activities
integrating clinical and basic science
and a longitudinal outpatient clinic
experience followed by debriefing in
postclinic groups. SDC is a case-based
curriculum, integrating clinical, basic,
and social sciences around patient
complaints and concerns across different
levels of organization and complexity.
Students need to synthesize and analyze
information that connects the different
concepts related to an activity. Mastery
of content knowledge is assessed using
progress examinations covering the
breadth of medical knowledge needed
for graduation, given each semester
of the 4-year curriculum.13 Given the
relationship between metacognition and
critical thinking,14 we were interested
in understanding this relationship
and how learners’ longitudinal growth
and performance growth can be
quantified through their progress
in the National Board of Medical
Examiners Comprehensive Basic Science
Examination (CBSE).
Previous studies have shown that
metacognitive awareness, critical
Academic Medicine, Vol. 96, No. 2 / February 2021
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
thinking, and learning strategies
influence students’ academic success.
More specifically, metacognitive
reading strategies have been found
to predict reading comprehension
among non-native English-speaking
medical students.15 Iranian medical
and dental students’ critical thinking
skills have been reported as significantly
correlated with numbers of passed
credits and passed research-methods
courses.16 Surgery residents’ learning
and studying strategies were found to
significantly affect their performance
on American Board of Surgery in-
training examinations.17 It has been
demonstrated that college biology
students use metacognition to evaluate
their learning strategies and learning
plans and are more sophisticated in
their regulation of metacognition if they
know about how people learn.18
In this study, we aimed to understand the
relationship between critical thinking,
metacognition, and self-regulation
and students’ progress test scores, as
well as how these relationships might
help us predict student performance at
different time points. Predicting students’
progress may help us identify students
at risk who would benefit from early
interventions by academic achievement
support. To our knowledge, most work
in this vein has been tested with either
undergraduate students or nursing
students.18–20 Only a few studies based on
medical students have assessed the impact
of critical thinking, metacognition,
and self-regulation on medical student
performance, and they did so separately
with end-point outcome using cross-
sectional designs.21,22 In addition, most
of the existing studies used summative
scores across categories or scales to
represent the specific latent construct,
ignoring the measurement error in
the scales, which can overestimate or
underestimate the effect of the true
underlying construct the observed scales
represent. This study expanded the scope
beyond that of the previous research
cited above. We proposed a framework
to account for the relationships
among the underlying constructs (e.g.,
metacognition) and estimated the
measurement errors in the observed
scales representing these constructs. We
also incorporated longitudinal data for
achievement performance to address
medical students’ growth in medical
knowledge over time.
Research Report
Method
Data sources and measurements
Participants in this study were184 first-year
medical students from the MSU–CHM
who matriculated in fall 2017. The study
was deemed exempt by the MSU Office
of Regulatory Affairs Human Research
Protection Programs.
We investigated participating students’
metacognitive knowledge and regulation
using the Metacognitive Awareness
Inventory (MAI)5 and critical thinking
skills as measured by the Watson-Glaser
Critical Thinking Appraisal (WG-
CTA).23 Students completed the MAI,
WG-CTA, and Learning and Study
Strategies Inventory (LASSI) in August
2017 at matriculation, and their medical
knowledge was assessed 5 times from
fall 2017 through spring 2019. The
strength of these measures is that they
were developed through professional
psychometricians. The psychometric
properties, scale development process,
scoring, and reporting strategies were
well documented and are available in
their technical reports.
Metacognition. The MAI,5 used to assess
metacognition, is a self-assessment with
a 5-point rating scale that includes 2
dimensions: knowledge of cognition
and regulation of cognition. Knowledge
of cognition includes 3 categories:
declarative knowledge, procedural
knowledge, and conditional knowledge.
Regulation of cognition includes 5
categories: planning, information
management, monitoring, debugging,
and evaluation. In total, the MAI includes
52 items. Coefficient α for the entire
MAI reached 0.95, and coefficient α
for items loading on each factor reached
0.91.5 Each category was measured by
4 to 9 questions on a scale of 1 to 5,
where 5 is the lowest score. We reverse
coded each question and summed
questions within each category for the
analysis in this study. To revise the MAI
for the population of medical students,
we conducted a factor analysis to
examine the indicator relations in the
measurement model. Based on the model
modification suggestion from the factor
analysis, we used the total knowledge of
cognition scores in the proposed model
in the study.
Critical thinking. We measured medical
students’ critical thinking ability
279
 Research Report
using the WG-CTA.23 The following 3
subcategories were used to construct
the model in this study: recognition of
assumptions, evaluation of arguments,
and drawing conclusions. Across multiple
forms, the internal consistency reliability
coefficients of the total scores are 0.81 to
0.83 and standard errors of measurement
are 2.63 to 2.64.24 We explored the
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
correlations between WG-CTA scores and
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/29/2023
other cognitive ability measures such as
SAT, ACT, and WAIS-IV scores to provide
evidence for the convergent validity. The
results showed that a strong association
exists especially with the measures of
reasoning.24
Self-regulation. Medical students’ self-
regulated learning strategies (SR) were
measured by one of the components of
LASSI.25 The self-regulation component
includes 4 scales: concentration, time
management, self-testing, and using
academic resources. The percentile scores
of these 4 scales were collected and used
in this study. The percentile scores were
based on the national norming sample
of 1,092 students from 12 different
institutions in the United States.25 The
Cronbach’s α s for the 4 scales are above
0.8.25 We examined and validated the scale
scores against performance measures.25
Progress examination performance.
We assessed medical students’ medical
knowledge 5 times, or once a semester,
using the CBSE26 from the National
Board Medical Examiners, which is the
only form of basic science summative
assessment administered in the SDC.
The CBSE was administered in fall 2017,
spring 2018, summer 2018, fall 2018,
and spring 2019. The item difficulties
of the questions were around 0.7. We
determined students’ performance by
their CBSE total scores over time.
Analysis
Structural equation modeling. We
investigated the relationships of
constructs under the latent variable
framework.27 The 5 longitudinal
performance scores from the CBSE
were modeled by 2 latent constructs: the
intercept (i.e., the starting point) and the
growth (i.e., the shape of the growth)
using a growth curve model. To model
the shape of growth proportion across 5
examinations, we fixed the estimate of the
first-time CBSE on the growth construct
as 0 and the estimate of the last CBSE
280
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
score on the growth construct as 1. The
growth estimates of the progress tests at
the second, third, and fourth time points
were estimated freely, and the values
represent the proportion of growth from
the first time point.
Metacognitive awareness, critical
thinking ability, and self-regulated
learning strategies are latent constructs
as well and measured by different
subscales, as mentioned above. To
consider their relationships with the
progress assessment results, we applied a
latent variable framework or structural
equation model (SEM) to incorporate
the 3 latent variables and single growth
curve model simultaneously into
a comprehensive model. Given the
small sample size in this study,28 we
followed the widely accepted rules for
10 subjects per indicator29 and used the
subcategories instead of questions in
scales as the indicators in the proposed
model to ensure adequate power. To
confirm the theory that there would be
intercorrelation between metacognition,
critical thinking, and self-regulation, the
3 latent constructs (i.e., MAI, WG-CTA,
and SR) were included in the model.
Statistical analysis procedure. We
first fit the 3 measurement models
individually to examine each measure,
then fit the growth curve models using
the 5 CBSE scores. Next, we combined
the measurement models and the growth
curve model to examine the structural
relationships and estimate the intercept
and growth of the growth curve model.
The data were collected at matriculation,
cleaned, merged, and deidentified by the
Honest Broker (a designated university
employee, independent of the research
team, that created the deidentified
dataset) in the MSU–CHM.30 We
summed the questions for subcategories
in each instrument and selected the
variables using R statistical software,
version 3.5.3 (R, Vienna, Austria) for
model fitting. The model was run using
Mplus software, version 8 (Muthén LK,
Muthén BO, Los Angeles, California).
Evaluation criteria. Although there is no
universal index in evaluating the model
fit for all kinds of SEMs, we followed the
consensus in the current literature31,32 that
suggested multiple indices combinational
rules for its acceptable cutoff criteria:
root-mean-square error of approximation
(RMSEA) of 0.07 or lower,33,34 a
Academic Medicine, Vol. 96, No. 2 / February 2021
standardized root-mean-square residual
(SRMR) of 0.09 or lower,31 and a Tucker–
Lewis index (TLI) of 0.90 and above35 for
the least sums of Type I and Type II error
rates for model evaluation. Since the χ
2
test statistic is sensitive to the sample size
and the complexity of models, choosing
RMSEA, SRMR, and TLI in combination
allowed the model to be evaluated fairly
since these indices are insensitive to
sample sizes, misspecified model, and the
complexity of the models.31–35
Results
Descriptive statistics for the indicators
used from the 184 participating students
are shown in Table 1. Using the multiple
indices combinational rules for model
evaluation,31–35 the model fit indices
showed that the proposed model had
a fair and adequate fit to the data
(RMSEA = 0.062 (90% confidence interval
[0.047, 0.076], P = .08); SRMR = 0.09;
TLI = 0.924). The estimates of the factor
loadings and the structural relationships
are shown in Tables 2 and 3. The SEM
diagram can be found in Supplemental
Digital Appendix 1, available at http://
links.lww.com/ACADMED/B29.36
The factor loadings of 3 latent variables
(i.e., critical thinking, self-regulation, and
metacognition) are shown in Table 2. All
indicators we chose to include showed
that they had significant contributions to
their factors.
If students’ 2-year growth is regarded
as 100%, the growth proportion in the
growth curve model showed that students’
growth in the first 2 semesters was about
31.0%. By the first summer, their growth
from the beginning was about 47.5%.
By the end of the fall, or 1.5 years after
matriculation, the growth was about
77.6%. Subtracting 77.6% from 100%,
the remaining 22.4% of growth occurred
in the final semester, before students
take the United States Medical Licensing
Examination Step 1. In other words, we
saw students’ growth accelerate at the
beginning (31.0%), slow down between
the second and the third time points
(about 16.5%, or 47.5% to 31.0%), and
then accelerate again in the last 2 semesters
before the Step 1 exam (30.1% and 22.4%
of growth, respectively, Figure 1).
The growth curve model of progress
performance showed no relationship
between students’ initial knowledge
 Research Report

metacognitive awareness, critical

Table 1 thinking, and self-regulated learning
Descriptive Statistics for Indicators in the Structural Equation Model, From a strategies, and to investigate the
Study of Examination Performance and Metacognition, Critical Thinking, and associations among them, as well as their
Self-Regulated Learning Strategies, Michigan State University College of Human relationships with medical students’
Medicine, 2017–2019
initial medical knowledge and growth
Latent construct in knowledge over their first 2 years of
and indicators Measurement tool Min. Max. Mean (SD) medical school. The findings suggest
that critical thinking is related to initial
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X

Critical thinking Watson-Glaser Critical

Thinking Appraisal test performance, but not to growth
in medical knowledge over time. The
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/29/2023

 Drawing conclusions 1 16 12.64 (2.40)

 Recognition of assumptions
 Evaluation of arguments
Self-regulation Learning and Study
Strategies Inventory
 Concentration
 Self-testing
 Time management
 Using academic resources
Metacognition Metacognitive Awareness
Inventory
 Knowledge of cognition
 Planning
 Information management
 Monitoring
 Debugging
 Evaluation
Intercept and growth Comprehensive Basic
Science Examination
 CBSE 1: Fall 2017
 CBSE 2: Spring 2018
 CBSE 3: Summer 2018
 CBSE 4: Fall 2018
 CBSE 5: Spring 2019
Abbreviations: Min., minimum; Max., maximum; SD, standard deviation; CBSE, Comprehensive Basic Science
Examination.
score and their growth. However, the 3
structural relationships were associated
with initial knowledge score and the
growth. Specifically, critical thinking was
positively correlated with students’ initial
knowledge score (β =1.956, P < .001).
Self-regulation had a significantly positive
relationship (β =3.287, P < .05), but
metacognition had a negative relationship
with the growth (β =−3.426, P < .01).
Medical students with better critical
thinking ability at matriculation tended
to score higher on their progress tests at
the beginning, but this ability did not
correlate with the growth of the progress
performance over time. On the other
hand, self-regulation scores had no
significant relationship with the initial
knowledge score (β =0.238, P > .05)
but had a positive relationship with the
growth in progress score in the first 2
years. Metacognition had no relationship
1 12 8.17 (3.01) level of self-regulation measured with
1 12 8.25 (1.91) LASSI had a positive relationship with
participants’ performance growth on the
CBSE, whereas the level of metacognition
1 99 60.54 (25.12)
had an inverse relationship with such
growth during the study period.
1 99 56.78 (23.99)
1 99 62.84 (27.31) These findings illustrate the trajectory of
1 99 52.53 (26.28) participating medical students’ growth
during the first 2 years of the SDC. This
trajectory describes students’ learning
17 85 68.45 (6.08) curve, quantifies the growth proportion
7 35 26.67 (3.57) at each semester, and plots the transition
9 45 36.61 (4.09) point on the timeline when students
7 35 28.06 (3.25) are preparing for their Step 1 exams.
To model students’ growth or progress,
5 25 21.69 (1.92)
concepts (or constructs) must be
5 25 18.30 (2.68)
measured repeatedly. Though designing
comparable constructs takes tremendous
effort, students’ learning trajectories
1 100 36.98 (5.08) can be examined, and at-risk students
1 100 47.20 (6.27) can be identified early. Furthermore, a
1 100 52.41 (7.85) longitudinal design usually has better
1 100 62.25 (10.07) power than a cross-sectional design.
1 100 70.13 (10.98) Using longitudinal designs can improve
the power of medical education research
since increasing the number of times
that concepts are measured is easier than
increasing the sample size.
with the initial score (β = 0.605, P > .05),
and higher scores in this area were In our findings, this logit pattern for
associated with slower growth in progress growth in medical knowledge (i.e.,
performance in the first 2 years. rapid growth, followed by a plateau
in performance, followed by rapid
The model also estimated the structural growth then a final deceleration)
relationships among critical thinking, seems reasonable in the context of
self-regulation, and metacognition. There several learning theories. In the first
was no significant relationship between stage, students learn new terminology
critical thinking and self-regulation, and begin to integrate prior science
nor between critical thinking and knowledge within a clinical context.
metacognition. The relationship between New knowledge resides in working
self-regulation and metacognition was memory and is immediately applied
significant (β =0.476, P < .001). Medical to the progress examinations. As the
students with higher scores for self- curriculum progresses, students must
regulation tended to score higher on consolidate knowledge to long-term
metacognition skills. memory while continuing to acquire
new knowledge to be stored within finite
working memory. In a discovery learning
Discussion context such as the SDC, students run the
This study used an SEM framework risk of cognitive overload, which could
to combine multiple models of result in the plateau of performance.

Academic Medicine, Vol. 96, No. 2 / February 2021 281

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
 Research Report

suggesting how medical students

Table 2 interacted with the environment and
Factor Loadings for the Measurement Models of 3 Latent Concepts in the Structural the curriculum during the first 2 years.
Equation Model, From a Study of Examination Performance and Metacognition, Our findings may allow researchers to
Critical Thinking, and Self-Regulated Learning Strategies, Michigan State University enhance situated learning theory with
College of Human Medicine, 2017–2019
longitudinal studies that model the
Latent variable and person–environment interaction over
indicators Measurement tool Estimate (SE) P value time. Findings also carry implications
for interventions to improve students’
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X

Critical thinking Watson-Glaser Critical

Thinking Appraisal progress performance. Faculty members
and academic specialists can promote
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/29/2023

 Drawing conclusions 1.926 (0.242) < .001

 Recognition of assumptions 1.477 (0.263) < .001 self-regulated learning strategies as a part
 Evaluation of arguments 0.871 (0.171) < .001 of the curriculum or through individual
coaching.
Self-regulation Learning and Study
Strategies Inventory
 Concentration 19.807 (2.163) < .001
For this study, we have used knowledge
of cognition as a summation of the 3
 Self-testing 9.470 (2.248) < .001
subscales of declarative, procedural,
 Time management 22.843 (2.350) < .001 and conditional knowledge, as a single
 Using academic resources 8.875 (2.469) < .001 indicator of the MAI. Knowledge of
Metacognition Metacognitive cognition has been shown to correlate
Awareness Inventory with exam performance for first-year
 Knowledge of cognition 4.174 (0.449) < .001 medical students, with little variation
 Planning 2.567 (0.261) < .001 among the 3 subscales.37 This contrasts
 Information management 2.923 (0.299) < .001 with the greater variation seen among
 Monitoring 2.595 (0.227) < .001 college students. Modifying the MAI
measure by combining these 3 subscales
 Debugging 1.045 (0.152) < .001
allowed us to increase the degrees
 Evaluation 1.921 (0.194) < .001
of freedom of the model without
Abbreviation: SE, standard error. compromising the contextual meaning of
the MAI measure.
The plateau also occurs at the end of a insights into previously encountered Similar to findings from previous
curricular segment (i.e., the Early Clinical problems. The final period of slowing studies,11,14 metacognition correlated
Experience, during which first-year may be due to the law of diminishing with self-regulation in this study.
students rotate in one ambulatory clinic); returns; students have spent considerable Metacognitive awareness did not correlate
students reaccelerate after entering a time consolidating medical knowledge with the growth of performance, but
new curricular segment (i.e., the Middle to long-term memory in the preceding self-regulation components measured
Clinical Experience, when second-year months and may receive less benefit by LASSI did correlate. These results
students rotate through 12 distinct from ongoing investment in studying. indicate that students who self-regulate
clinical settings). The new settings Whatever the underlying cause, our and develop effective strategies such as
may support reinvigorated learning findings provide empirical evidence that planning, information management,
as students’ contexts and experiences can help quantify students’ expected monitoring, debugging, and evaluation
change, providing new viewpoints and trajectory of progress performance, do not necessarily require a sophisticated
understanding of their own cognitive
processes.
Table 3
Structural Relationships (Covariance Matrix) Among Latent Variables in the The inverse relationship between
Structural Equation Model, From a Study of Examination Performance and
Metacognition, Critical Thinking, and Self-Regulated Learning Strategies, Michigan metacognitive awareness and progress
State University College of Human Medicine, 2017–2019a performance has been observed in
previous studies with different study
Latent Critical populations. Sperling and colleagues38
variable Intercept Growth thinking Self-regulation
found a similar negative relationship
Intercept — — — — between MAI and SAT mathematics
Growth 1.029 (3.940) — — — scores among high school students.
Critical thinking 1.956 (0.438)b 1.639 (0.970) — — Roohani and Khalilian39 studied English
Self-regulation 0.238 (0.575) 3.287 (1.279) c
−0.120 (0.113) — as foreign language learners and found
Metacognition 0.605 (0.506) −3.426 (1.144)d −0.108 (0.101) 0.476 (0.090)b a negative relationship between MAI
scores and their vocabulary skill.
a
The value in parentheses is standard error. In all 3 populations, foundational
b
P < .001.
c
P < .05. knowledge focusing on recall, applying,
d
P < .01. and analyzing, rather than higher-

282 Academic Medicine, Vol. 96, No. 2 / February 2021

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Figure 1 Growth proportion of 184 participating medical students in progress performance in their first 2 years, from a study of examination performance
and metacognition, critical thinking, and self-regulated learning strategies, Michigan State University College of Human Medicine, 2017–2019. Students’
growth in medical knowledge showed rapid growth at the beginning from fall 2017 to spring 2018, followed by a plateau in performance (16.50%
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
growth from spring 2018 to summer 2018), followed by rapid growth (30.10%), and a final deceleration (22.40%) in spring of the second year.
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/29/2023
level cognitive skills, maybe in play.
Nevertheless, we intend to explore the
relationship between the MAI constructs
of knowledge of cognition and regulation
of cognition when more students
have completed the first 2 years of our
curriculum, thereby increasing our
sample size.
Methodologically, our modeling
approach accounted for the measurement
error of the 3 scales measuring critical
thinking, self-regulated learning
strategies, and metacognitive awareness;
solved the multiple-testing issues that we
believe affected the findings of previous
studies; and provided rigorous findings.
However, except for CBSEs, none of the 3
instruments we chose were developed for
medical students. In addition, our small
sample size also required us to exclude
participant demographic information
from the SEM. With a larger sample
size, we could use 1 of 2 approaches
to examine the relationship between
demographic and latent variables. The
first would be to conduct the Multiple
Indicators Multiple Causes model, in
which demographic variables are used
to contextualize the latent variables.
In this model, estimates represent a
controlled relationship between the 2
variables, which may make interpretation
difficult. The second option would be the
multiple-group approach SEM, which
allows investigation of differences of
construct relationships across categories
(e.g., ethnic or sex groups). This model
requires large sample sizes; the number
of to-be-estimated parameters increases
as the number of to-be-compared groups
increases.
We suggest that our empirical findings
can be beneficial to the study of learning
strategies and can provide direction and
guidance for medical educators. Coaches
and tutors can now provide interventions
at the time points when students need it
the most. For example, critical thinking
strategies could be provided to students
Academic Medicine, Vol. 96, No. 2 / February 2021
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
just after matriculation and before
the first CBSE exam. Self-regulation
workshops could be provided during
the first 2 years if the progress scores are
not satisfying. We can establish an early
warning system to keep track of students’
performance in progress examinations
and intervene when they are not on track.
The interventions, resources, and activities
can be customized to meet students’ needs
based on their trajectories of student
performance in progress examinations.
For example, if students are struggling at
the beginning of medical school, we could
help them practice critical thinking skills.
Focusing on the skills that help them to
conceptualize, apply, analyze, synthesize,
and evaluate the information they learn
at school will empower students to better
understand and digest the materials and
information they receive. If the students’
growth trajectories in progress test
performance over time slow down, we
may place more emphasis on practicing
effective habits using self-regulated
learning strategies such as goal setting,
reflective dialogue practice, and phrase
feedback about tasks. In conclusion,
student performance on standardized
progress examinations together with the
WG-CTA, MAI, and LASSI may facilitate
understanding of better individual
learning strategies and design of
interventions to promote student success
in an integrated curriculum.
Acknowledgments: The authors would like
to thank Ann Taft in the Office of Medical
Education Research and Development for her
support of data cleaning and the roundtable
audiences for comments and feedback when the
original manuscript was presented at the annual
conference of the 2019 American Educational
Research Association. The authors would also like
to thank Dr. Renoulte Allen for facilitating the
administration of the Metacognitive Awareness
Inventory, Watson-Glaser Critical Thinking
Appraisal, and Learning and Study Strategies
Inventory to the cohort of participating students.
Funding/Support: Lilly Teaching Fellows Program
from Michigan State University.
Other disclosures: None reported.
Research Report
Ethical approval: Project reviewed by the
Michigan State University Office of Regulatory
Affairs Human Research Protection Programs
(IRB# x17-961e Category: Exempt 2 Approval
Date: July 19, 2017).
Previous presentations: This article is extended
from a conference paper presented at the Annual
Conference of the American Educational
Research Association, Toronto, Canada, in April
7, 2019.
C. Chang is assistant professor, Office of Medical
Education Research and Development and
Department of Epidemiology and Biostatistics,
College of Human Medicine, Michigan State
University, East Lansing, Michigan; ORCID: https://
orcid.org/0000-0002-2106-2476.
M. Colón-Berlingeri is assistant professor,
Department of Physiology, College of Natural
Science, Michigan State University, East Lansing,
Michigan.
B. Mavis is professor, Office of Medical Education
Research and Development, College of Human
Medicine, Michigan State University, East Lansing,
Michigan; ORCID: https://orcid.org/0000-0003-
2145-3634.
H.S. Laird-Fick is professor, Department of
Medicine, and director, Office of Assessment,
College of Human Medicine, Michigan State
University, East Lansing, Michigan; ORCID: https://
orcid.org/0000-0001-9215-8152.
C. Parker is assistant dean, Program Evaluation and
Continuous Quality Improvement, Office of Medical
Education Research and Development, College of
Human Medicine, Michigan State University, East
Lansing, Michigan.
D. Solomon is professor emeritus, Department of
Medicine and Office of Medical Education Research
and Development, College of Human Medicine,
Michigan State University, East Lansing, Michigan;
ORCID: https://orcid.org/0000-0002-3130-5240.
References
1 Tanner KD. Promoting student
metacognition. CBE Life Sci Educ.
2012;11:113–120.
2 Dunlap JC, Grabinger S. Preparing
students for lifelong learning: A review
of instructional features and teaching
methodologies. Perform Improv Q.
2008;16:6–25.
3 Li ST, Paterniti DA, Co JP, West DC.
Successful self-directed lifelong learning
in medicine: a conceptual model derived
from qualitative analysis of a national
survey of pediatric residents. Acad Med.
2010;85:1229–1236.
283
 Research Report
4 Dawson TL. Metacognition and learning in
adulthood. White paper. ODNI/CHO/IC
Leadership Development Office. 2008:1–22.
5 Schraw G, Dennison RS. Assessing
metacognitive awareness. Contemp Educ
Psychol. 1994;19:460–475.
6 Zimmerman BJ. Self-regulated learning and
academic achievement: An overview. Educ
Psychol. 1990;25:3–17.
7 Pintrich PR. The role of metacognitive
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
knowledge in learning, teaching, and
assessing. Theory Pract. 2002;41:219–225.
8 Morra S, Gobbo C, Marini Z, Sheese R.
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/29/2023
Cognitive Development: Neo-Piagetian
Perspectives. London, UK: Psychology Press;
2007.
9 Piccinini G, Scarantino A. Information
processing, computation, and cognition. J
Biol Phys. 2011;37:1–38.
10 Hanks WF. Foreword. In: Lave J, Wenger E,
eds. Situated Learning: Legitimate Peripheral
Participation. Cambridge, UK: Cambridge
University Press; 1991:14.
11 Lajoie SP. Metacognition, self regulation, and
self-regulated learning: A rose by any other
name? Educ Psychol Rev. 2008;20:469–475.
12 Medina MS, Castleberry AN, Persky
AM. Strategies for improving learner
metacognition in health professional
education. Am J Pharm Educ. 2017;81:78.
13 Michigan State University College of Human
Medicine. SCRIPT and the Shared Discovery
Curriculum. https://curriculum.chm.msu.
edu/curricular-content/script-competencies.
Published 2013. Accessed September 7,
2020.
14 Magno C. The role of metacognitive skills in
developing critical thinking. Metacognition
Learn. 2010;5:137–156.
15 Shang H-F. EFL medical students’
metacognitive strategy use for hypertext
reading comprehension. J Comput High
Educ. 2018;30:259–278.
16 Mahmoodabad SS, Nadrian H, Nahangi H.
Critical thinking ability and its associated
factors among preclinical students in Yazd
Shaheed Sadoughi University of Medical
Sciences (Iran). Med J Islam Repub Iran.
2012;26:50–57.
284
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
17 Derossis AM, Da Rosa D, Schwartz A, Hauge
LS, Bordage G. Study habits of surgery
residents and performance on American
Board of Surgery in-training examinations.
Am J Surg. 2004;188:230–236.
18 Stanton JD, Dye KM, Johnson M. Knowledge
of learning makes a difference: A comparison
of metacognition in introductory and senior-
level biology students. CBE Life Sci Educ.
2019;18:ar24.
19 Behrens PJ. The Watson-Glaser Critical
Thinking Appraisal and academic
performance of diploma school students. J
Nurs Educ. 1996;35:34–36.
20 Jouhari Z, Haghani F, Changiz T. Assessment
of medical students’ learning and study
strategies in self-regulated learning. J Adv
Med Educ Prof. 2016;4:72–79.
21 Hong WH, Vadivelu J, Daniel EGS, Sim JH.
Thinking about thinking: Changes in first-
year medical students’ metacognition and its
relation to performance. Med Educ Online.
2015;20:27561.
22 Turan S, Demirel O, Sayek S. Metacognitive
awareness and self-regulated learning skills
of medical students in different medical
curricula. Med Teach. 2009;31:e477–e483.
23 Watson G, Glaser EM. Watson-Glaser Critical
Thinking Appraisal: Manual. London, UK:
Psychological Corp; 2002.
24 Watson G, Glaser EM. Watson-Glaser II
Critical Thinking Appraisal: Technical
Manual and User’s Guide. San Antonio, TX:
Pearson; 2010.
25 Weinstein CE, Palmer DR, Acee TW. LASSI
User’s Manual. Clearwater, FL: H & H
Publishing Company; 2016.
26 National Board of Medical Examiners.
Comprehensive Basic Science
Examination. https://www.nbme.org/
Schools/Subject-Exams/Subjects/comp_
basicsci.html. Published 2018. Accessed
September 9, 2020.
27 Raykov T, Marcoulides GA. A First Course
in Structural Equation Modeling. 2nd ed.
Mahwah, NJ: Lawrence Erlbaum; 2006.
28 Wolf EJ, Harrington KM, Clark SL, Miller
MW. Sample size requirements for structural
equation models: An evaluation of power,
Academic Medicine, Vol. 96, No. 2 / February 2021
bias, and solution propriety. Educ Psychol
Meas. 2013;76:913–934.
29 Nunnally JC. Psychometric Theory. New
York, NY: McGraw-Hill; 1967.
30 Michigan State University College of Human
Medicine. Office of Medical Education
Research and Development. Honest Broker
for Educational Scholarship. https://omerad.
msu.edu/research/honest-broker-for-
educational-scholarship. Accessed September
9, 2020.
31 Hu LT, Bentler PM. Cutoff criteria for fit
indexes in covariance structure analysis:
Conventional criteria versus new alternatives.
Struct Equ Model. 1999;6:1–55.
32 Hooper D, Coughlan J, Mullen MR.
Structural equation modelling: Guidelines
for determining model fit. Electron J Bus Res
Methods. 2008;6:53–60.
33 Steiger JH. Structural model evaluation
and modification: An interval estimation
approach. Multivariate Behav Res.
1990;25:173–180.
34 Steiger JH. Understanding the limitations of
global fit assessment in structural equation
modeling. Pers Individ Dif. 2007;42:893–898.
35 Sharma S, Mukherjee S, Kumar A, Dillon
WR. A simulation study to investigate the
use of cutoff values for assessing model fit
in covariance structure models. J Bus Res.
2005;58:935–943.
36 MacCallum RC, Browne MW, Sugawara HM.
Power analysis and determination of sample
size for covariance structure modeling.
Psychol Methods. 1996;1:130–149.
37 Welch P, Young L, Johnson P, Lindsay D.
Metacognitive awareness and the link with
undergraduate examination performance
and clinical reasoning. MedEdPublish.
https://www.mededpublish.org/
manuscripts/1603. Published 2018. Accessed
September 9, 2020.
38 Sperling RA, Howard BC, Staley R, DuBois
N. Metacognition and self-regulated learning
constructs. Educ Res Eval. 2004;10:117–139.
39 Roohani A, Khalilian H. On the relationship
between metacognitive awareness and two
aspects of vocabulary knowledge. Iran J Appl
Linguist. 2012;15:97–122.