Journal of Personality Assessment

ISSN: (Print) (Online) Journal homepage: www.tandfonline.com/journals/hjpa20

Psychometric Properties of the MMPI-3

in a Sample of Black and White American
Undergraduate Students: Examining Group
Differences and Convergent/Discriminant Validity
with the ASEBA Adult Self-Report

Joye C. Anestis, Taylor R. Rodriguez, Chloe O’Dell, Olivia C. Preston, Tiffany M.

Harrop & Nora E. Charles

To cite this article: Joye C. Anestis, Taylor R. Rodriguez, Chloe O’Dell, Olivia C. Preston, Tiffany
M. Harrop & Nora E. Charles (2024) Psychometric Properties of the MMPI-3 in a Sample
of Black and White American Undergraduate Students: Examining Group Differences and
Convergent/Discriminant Validity with the ASEBA Adult Self-Report, Journal of Personality
Assessment, 106:1, 1-16, DOI: 10.1080/00223891.2023.2214817

To link to this article: https://doi.org/10.1080/00223891.2023.2214817

View supplementary material Published online: 30 May 2023.

Submit your article to this journal Article views: 483

View related articles View Crossmark data

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=hjpa20
Journal of Personality Assessment
2024, VOL. 106, NO. 1, 1–16
https://doi.org/10.1080/00223891.2023.2214817

Psychometric Properties of the MMPI-3 in a Sample of Black and White

American Undergraduate Students: Examining Group Differences and
Convergent/Discriminant Validity with the ASEBA Adult Self-Report
Joye C. Anestis1 , Taylor R. Rodriguez1,2 , Chloe O’Dell3 , Olivia C. Preston3 , Tiffany M. Harrop4
and Nora E. Charles3
1
Rutgers University; 2New Jersey Gun Violence Research Center; 3University of Southern Mississippi; 4Missouri Department of Mental Health

ABSTRACT ARTICLE HISTORY

The current study examined MMPI-3 internal and external psychometric properties with a focus Received 16 March 2023
on the impact of racialized group membership (Black and White Americans). The second aim was Revised 28 April 2023
to examine convergent/discriminant MMPI-3 scale associations with a different broadband, Accepted 3 May 2023
hierarchical self-report assessment tool [Adult Self-Report (ASR)]. Consistent with findings on prior
MMPI iterations, we expected to observe no clinically meaningful mean differences on MMPI-3
scale T-scores. We hypothesized that validity coefficients between MMPI-3 and ASR scales measuring
similar constructs would be stronger (convergent validity) and the inverse for scales measuring
disparate constructs (discriminant validity). We also expected coefficient magnitude consistency
across racial groups. The final sample was composed of 254 undergraduates (74.4% female; 63.8%
White, 36.2% Black). Results suggest 1) MMPI-3 substantive scale mean T-scores are comparable
between White and Black American undergraduates; 2) MMPI-3 scales correlate with ASR scale
scores in expected ways with regard to internalizing problems, rule breaking and impulsivity,
thought problems, and substance use (but not overall externalizing, aggression, attention problems,
and intrusiveness); and 3) convergent and discriminant associations between MMPI-3 and ASR
scales are consistent across White and Black Americans. This work provides support for MMPI-3
use with racially diverse individuals, considers next steps for understanding MMPI-3 scale score
functioning in diverse populations, and provides novel information on MMPI-3 correspondence
with the ASR.

The Minnesota Multiphasic Personality Inventory-3
(MMPI-3) is the most recent iteration of the MMPI family
of instruments, which continue to be amongst the most
widely utilized measures of personality and psychopathology
(Ben-Porath & Tellegen, 2020a, 2020b; Camara et al., 2000;
Sellbom, 2019; Wright et al., 2017). The MMPI-3 retains
the hierarchical-dimensional organization implemented in
the MMPI-2-RF but also features an updated and contem-
porary normative sample designed to align with the socio-
demographic composition of the United States (U.S.), new
or rewritten items (≈30% of items), five new scales, as well
as updates to retained scales (>90% of scales were retained).
Although the MMPI-3 stands on an extensive empirical
foundation (see Ben-Porath & Tellegen, 2020a; Sellbom,
2019, for a review), several areas of research require addi-
tional focus, including exploring linkages to other
hierarchical-dimensional clinical assessment tools and com-
prehensive empirical examinations of sociodemographic (e.g.,
race, gender) group differences on test scores. The current
study aims to contribute knowledge in both domains by
examining MMPI-3 internal and external properties (i.e.,
associations with other hierarchical-dimensional models)
across Black and White Americans.
Consideration of racialized group differences in the
MMPI instruments
Contemporary assessment work has emphasized the impor-
tance of empirically testing the assumption that psycholog-
ical tests can be used across diverse populations, especially
since demographic-based biases can lead to erroneous diag-
noses, false conclusions, and negative consequences for
groups that are already discriminated against (e.g., Bagby
et al., 2022; Han et al., 2019). A substantial strength of the
MMPI-3 test development process is the acquisition of a
new, contemporary normative sample recruited to approxi-
mate 2020 US Census projections with respect to race, edu-
cation, and age (Ben-Porath & Tellegen, 2020a). As a result
of these efforts, MMPI-3 norms are based on a sample that
is 60.3% White, relative to the MMPI-2 normative sample
which was 81.8% White (Ben-Porath & Tellegen, 2020a;
Butcher et al., 1989). Confidence in the applicability of the

CONTACT Joye C. Anestis ja1206@sph.rutgers.edu School of Public Health, Rutgers University, 683 Hoes Lane West, Piscataway, NJ 08854, USA.
Supplemental data for this article can be accessed online at https://doi.org/10.1080/00223891.2023.2214817.
© 2023 Taylor & Francis Group, LLC
2 ANESTIS ET AL.
MMPI-3 to the greater US population is thus stronger than
that of the MMPI-2 and MMPI-2-RF, yet deliberate research
on internal and external validity properties of the MMPI-3
across diverse population, contexts, and settings
remain needed.
Validation samples used to examine psychometric prop-
erties are limited by the demographic characteristics of the
study setting and location. For example, the MMPI-3 test
developers recruited a large college student sample (n = 1546)
to identify empirical correlates of MMPI-3 scales (Ben-Porath
& Tellegen, 2020a). College student samples are an essential
and traditional component of test development and research;
students complete the large batteries of assessment measures
that are crucial for constructing validity and that would not
be permissible or feasible in most clinical settings. Further,
college student populations are experiencing an increased
prevalence of mental disorders (Auerbach et al., 2018), mak-
ing them appropriate participants in psychological assess-
ment research. Most college student samples are limited by
an overrepresentation of White participants, however, and
such is the case of the MMPI-3 college validation sample
which includes only about 151 Black students within a sam-
ple of 1546 participants (Ben-Porath & Tellegen, 2020a).
This highlights the need to examine MMPI-3 construct
validity evidence in other independent, geographically, and
demographically diverse settings, including amongst students
in college and universities with higher enrollments of racially
diverse students. The importance of this is paramount given
the role that empirical correlates play in the clinical realm,
namely providing the evidentiary support for scale inter-
pretive statements.
In the psychology research context, race typically func-
tions as a proxy for other constructs such as race-related
stressors, cultural values, and racial identity. Research
demonstrates that race-related stressors (e.g., discrimination,
systemic inequality) can negatively impact mental health
(see Williams, 2018, for a review), yet the impact of such
stressors on people of color is heterogeneous and thought
to be dependent on degree of ethnic/racial identity which
may buffer negative impacts or compound vulnerabilities
(Yip, 2018). At the same time, evidence suggests that psy-
chologists favor cultural over psychological explanations for
the behavior and cognitions of people of color and hold
the opposite bias for White people [the cultural (mis)attri-
bution bias; Causadias et al., 2018]. Higher scores obtained
on measures of psychological distress amongst members of
marginalized groups may reflect the impact of systemic
discrimination and an appropriate psychological response
to injustice as opposed to inherent individual-level psycho-
pathology. Investigating whether there is empirical support
for group-level score differences can aid in developing con-
siderations for clinical interpretations of individuals’ MMPI-3
profiles in practice.1
1
We note here that, whereas the focus of the current work is on racialized
groups [by which we mean socially constructed (i.e., not biological)
groups chiefly based on physical and behavioral characteristics (e.g., skin
color; Markus, 2008)], many intersecting aspects of an individual’s back-
ground and experience can plausibly influence assessment. Focusing on
Research on prior iterations of MMPI instruments, how-
ever, has largely not identified meaningful differential mean
scores or validity coefficients across racialized groups.
Moreover, in instances where prediction bias has been iden-
tified, it has been in the direction of slight underprediction
for Black Americans (i.e., intercept bias; Arbisi et al., 2002;
Marek et al., 2015; Timbrook & Graham, 1994; see Monnot
et al., 2009, for an exception). Two studies indicate that
MMPI-2-RF scales demonstrate sufficiently congruent pre-
dictive validity across sociodemographic groups (Marek
et al., 2015; Whitman et al., 2019), suggesting that the
scales are well-suited for predicting clinical outcomes
regardless of sociodemographic identities. This has been
established for many of the MMPI-2-RF Validity and
Substantive scales in a sample of bariatric surgery candi-
dates across gender, race, ethnicity, and age (Marek et al.,
2015) and in a forensic psychiatric sample across ethnicity
and gender (Whitman et al., 2019). While both studies
found some evidence of race-based bias, such differences
were ultimately deemed clinically insignificant due to the
negligible/small-sized effects (Marek et al., 2015; Whitman
et al., 2019). Similarly, in a large forensic mental health
clinic sample, Anestis et al. (2019) found that
MMPI-2-RF-measured psychopathic traits largely (i.e., 90.3%
of the time) manifested equivalently-sized correlations with
extra-test criteria across White and Black participants.
Taken together, research suggests that MMPI-2-RF scales
predict relevant outcomes similarly across race-based
groups, and these conclusions are thought to extrapolate
to the MMPI-3 (Ben-Porath & Tellegen, 2020a).
Most prior research in this area has been interested in
examining predictive bias (slope and intercept) via moder-
ated multiple regression. Extant literature, however, suggests
that such conclusions should be questioned, as they may be
due to methodological and statistical artifacts inherent in
the moderated multiple regression approach (see Aguinis
et al., 2010, for a comprehensive review). Close examination
of correlations alone is important as these values (i.e., valid-
ity coefficients derived from correlations) heavily inform test
interpretive statements and thus have direct clinical import.
Such an approach can be seen in Handel et al. (2011),
wherein correlation coefficient magnitudes between
MMPI-Adolescent (MMPI-A; Butcher et al., 1993) scales
and external criteria were examined across gender in an
adolescent forensic sample. At this time, we are aware of
only one published MMPI-3 study that has considered test
performance across racialized groups. Anestis et al. (2022)
examined MMPI-3 scale associations with reactions to a
behavioral task that elicited rejection, finding generally con-
sistent associations across White participants and participants
of color (majority Black). Notably, the race-based samples
were too small to examine differences in correlation mag-
nitudes, and the authors call for future research in this vein.
one aspect of someone’s identity (i.e., race) is only a preliminary step
in bolstering confidence in the use of assessment across diverse groups.
 MMPI-3 IN BLACK AND WHITE AMERICAN UNDERGRADUATES 3

Table 1. Hypothesized convergent/discriminant associations between MMPI-3 and ASR scales.

Adult Self-Report (ASR)
Mean
Anxious- Somatic Aggressive Rule Thought Attention Substance
MMPI-3 Scales Internalizing Depressed Withdrawn Complaints Externalizing Behavior Breaking Intrusive Problems Problems Use
Higher-Order (H-O)
EID (Emotional/Internalizing + + +
Dysfunction)
THD (Thought Dysfunction) +
BXD (Behavioral/ + + + + + +
Externalizing
Dysfunction)
Restructured Clinical (RC)
RCd (Demoralization) + + +
RC1 (Somatic Complaints) +
RC2 (Low Positive + + +
Emotions)
RC4 (Antisocial Behavior) + + + + + +
RC7 (Dysfunctional + + +
Negative Emotions)
RC6 (Ideas of Persecution) +
RC8 (Aberrant Experiences) +
RC9 (Hypomanic Activation) + + + + +
Specific Problems (SP)
Somatic/Cognitive
MLS (Malaise) +
NUC (Neurological +
Complaints)
EAT (Eating Concerns) +
COG (Cognitive Complaints) + +
Internalizing
SUI (Suicidal/Death + + +
Ideation)
HLP (Helplessness/ + + +
Hopelessness)
SFD (Self-Doubt) + + +
NFC (Inefficacy) + + +
STR (Stress) + +
WRY (Worry) + +
CMP (Compulsivity) + +
ARX (Anxiety-Related + +
Experiences)
ANP (Anger Proneness) + + + +
BRF (Behavior-Restricting + +
Fears)
Externalizing
FML (Family Problems) + + +
JCP (Juvenile Conduct + + +
Problems)
SUB (Substance Abuse) + + +
IMP (Impulsivity) + + + + + +
ACT (Activation) + + + + +
AGG (Aggression) + + +
CYN (Cynicism) + + +
Interpersonal
SFI (Self-Importance) +
DOM (Dominance) +
DSF (Disaffiliativeness) +
SAV (Social Avoidance) +
SHY (Shyness) +
Note. + denotes convergent validity hypothesis. Blank cells denote discriminant validity hypothesis.

Convergent and discriminant validity with other
Hierarchical-Dimensional psychopathology measures
The MMPI-3 retains the hierarchical-dimensional organi-
zation and interpretive framework developed for the
MMPI-2-RF and designed to align the instrument with
contemporary theory and knowledge on the structure of
psychopathology and personality (Ben-Porath & Tellegen,
2020b, 2020a). Despite these important goals, limited
research is available on the actual alignment of the MMPI-3
hierarchical domains with other hierarchical-dimensional
tools commonly used in clinical assessment. Arguably the
most prominent of these is the Achenbach System of
Empirically Based Assessment (ASEBA) which offers a mul-
tifaceted and hierarchical picture of adaptive and maladap-
tive functioning (Achenbach et al., 2017). The ASEBA
instruments have been well-validated in large, normative,
and cross-cultural samples and have been extended across
the lifespan, down to age 1.5 and up to age 90 (Achenbach
et al., 2017; Rescorla & Achenbach, 2004). Like the MMPI-3,
the ASEBA forms feature a structure with higher-order
4 ANESTIS ET AL.
Internalizing and Externalizing scales and lower-order syn-
drome scales, which should have theoretically expected
differential associations with the MMPI-3 scales (see Table
1). We are unaware of extant literature examining corre-
spondence between contemporary MMPI iterations and
ASEBA instruments, including across sociodemographic
groups. Correlations between ASEBA scales and MMPI-2
Clinical Scales are reported in the ASEBA manual, taken
from a small (n = 70) sample of Turkish undergraduate
students, but the substantial differences between the
MMPI-2 Clinical Scales and the Restructured Clinical Scales
(RC) used in the MMPI-3 render these correlations min-
imally useful in examining the construct validity evidence
of the MMPI-3. Handel et al. (2011) reported correlations
between MMPI-A Clinical, Content, and Supplementary
scales and both self- and caregiver-report ASEBA scales in
a large adolescent forensic sample, but again test differences
render comparisons challenging. To address these gaps and
advance knowledge of the linkage between MMPI con-
structs and those in other hierarchical tools, the current
study used the ASEBA Adult Self-Report (ASR) scales as
external criterion measures on which to examine convergent
and discriminant validity for and between Black Americans
and White Americans.
The current study
In sum, the current study was designed to investigate the
internal psychometric properties of the MMPI-3 Higher-Order
(H-O), RC, and Specific Problems (SP) scales as well as the
external psychometric properties of these scales in a sample
of Black and White American college students. While there
is ample literature demonstrating that lived experiences
among individuals of diverse racial identities can lead to
deleterious mental health impacts (Williams, 2018), MMPI
research to date suggests that mere membership in a racial
group does not impact test scores. As such, we hypothesized
that when examining descriptive statistics there would be
no meaningful differences (defined as ≥5 T-score points;
Graham et al., 2022) between MMPI-3 scale scores among
Black and White participants. The second aim of this
research was to examine convergent/discriminant MMPI-3
scale associations with a different broadband, hierarchical
self-report assessment tool (ASR). Specifically, we expected
convergent/discriminant validity would be identified such
that MMPI-3 and ASR scales assessing similar/convergent
constructs [e.g., MMPI-3 RC7 (Dysfunctional Negative
Emotions)-ASR Anxious-Depressed] would exhibit stronger
correlations than those assessing different/divergent con-
structs [e.g., MMPI-3 RC1 (Somatic Complaints)—ASR
Rule-Breaking Behavior]. Specific hypotheses for the con-
vergent/discriminant validity analyses were based on inter-
pretive recommendations for both the MMPI-3 and ASR
and are identified in Table 1. Additionally, given the lack
of predictive bias identified with prior MMPI iterations, we
hypothesized that the magnitude of convergent/discriminant
associations would be consistent across across White and
Black participants.
Methods
Participants
We began with a sample comprised of 3442 undergraduate
students at a mid-sized Southeastern university. Fifty-one
(14.8%) participants were removed based on MMPI-3 valid-
ity data [Cannot Say (CNS) >15; Combined Response
Inconsistency (CRIN), Variable Response Inconsistency
(VRIN), and True Response Inconsistency (TRIN) ≥80 T;
Infrequent Responses (F) and Infrequent Psychopathology
Responses (Fp) ≥100 T), consistent with established research
practices (e.g., Sellbom, 2021; Sellbom et al., 2021). Nine
(2.6%) were removed because they did not complete the
MMPI-3 due to technical problems. Of the remaining 284
participants, only individuals who identified as Black/African
American alone or who indicated Black/African American
as one of their multiple identities were included in the Black
subsample (n = 92; 32.4%) and those who identified only as
White in the White subsample (n = 162, 57.0%). The remain-
ing participants who reported a racial and/or ethnic group
identity outside of these definitions were excluded from
analyses (n = 30; 10.6%).
After these exclusions, the study sample (n = 254; 63.8%
White, 36.2% Black) had an average age of 21.38 (standard
deviation = 5.63; range = 18–51). The majority identified
as female/women regarding sex (n = 189, 74.4%) and gender
identity (n = 185; 72.8%). The majority also identified as
heterosexual (73.2%; n = 186). Nearly all participants were
full-time students (n = 242; 95.3%) and most were also
involved in paid work (n = 145; 57.1%). Detailed sociode-
mographic information by race group is reported in
Supplemental Table S1. Black and White participants did
not differ on sociodemographic characteristics, except Black
participants had proportionally more females/women and
proportionally less males/men than the White sample (sex:
φ = 0.26, gender: φ = 0.26).
Measures
Minnesota Multiphasic Personality Inventory-3 (MMPI-3;
Ben-Porath & Tellegen, 2020b, 2020a)
The MMPI-3 is a multiscale, self-report questionnaire of
psychopathology and personality characteristics organized
into a hierarchical-dimensional structure and assessing mal-
adaptive Emotional/Internalizing, Somatic/Cognitive,
Thought, Externalizing, and Interpersonal traits and behav-
iors. The instrument contains 335 items and features true/
false responding. For the purposes of the current study, only
the H-O, RC, and SP scales were considered. Descriptive
statistics for the MMPI-3 Substantive Scales are reported in
Table 2, including internal consistency reliability coefficients
(alpha and omega; Hayes & Coutts, 2020; McNeish, 2018).
All values were commensurate with ranges reported for the
normative samples in the MMPI-3 Technical Manual [α3:
2
A portion of this sample was included in Anestis et al. (2022).
3
We note that the Specific Problems scales such as STR, BRF, and EAT
are intentionally brief (i.e., 5-7 items), and scale length impacts (i.e.,
lowers) internal consistency reliability estimates.

White, 0.44 Stress (STR) − 0.92 Emotional/Internalizing
Dysfunction (EID). Black, 0.38 Stress (STR) − 0.88 EID; ω:
White, 0.55 Behavior Restricting Fears (BRF) − .92 EID.
Black, 0.43 Eating Concerns (EAT) − 0.90 EID.
Intercorrelations amongst the MMPI-3 H-O, RC, and SP
scales are reported in Supplemental Table S2. Participants
completed the computerized MMPI-2-RF-Expanded Form
(MMPI-2-RF-EX) from which MMPI-3 scale scores were
derived. Extant data indicate psychometric equivalence of
the two forms (Hall et al., 2022).
Adult self-report (ASR; Achenbach & Rescorla, 2003)
The ASR, part of the ASEBA taxonomy, is a 120-item
self-report measure designed to assess adult behavior by
asking participants to provide information about themselves.
Participants are prompted to report their behavior, thoughts,
and feelings in the past 6 months on a 3-point Likert scale
(0 = not true to 2 = very true). ASR items are distributed over
eight syndrome scales, a structure which has been replicated
globally (Ivanova et al., 2015). The combination of the
Anxious/Depressed (18 items), Withdrawn (9-items), and
Somatic Complaints (12 items) syndrome scales generate a
broadband Internalizing problems scale. The combination
of the Aggressive Behavior (15 items), Rule-breaking
Behavior (14 items) and Intrusive Behavior (6 items) syn-
drome scales generate a broadband Externalizing problems
scale. The final two syndrome scales are Attention Problems
(15 items) and Thought Problems (10 items). Per the ASR
manual, raw scores were generated for broadband and syn-
drome scales and subsequently used for analyses. Apart from
these scores, a Mean Substance Use total score was also
generated and used in analyses in accordance with manual
protocol. ASR descriptive statistics and internal consistency
reliability coefficients are reported in Supplemental Table
S3 and ASR intercorrelations are reported in Supplemental
Table S4. Except for the Intrusive scale in the Black sample
(α = 0.56, ω = 0.59), all values were consistent with those
reported in the test manual [α: White, 0.69 (Intrusive) −
0.94 (Internalizing). Black, 0.60 (Thought Problems) − 0.92
(Internalizing). ω, White, 0.67 (Intrusive) − 0.94
(Internalizing). Black, 0.74 (Withdrawn) − 0.92
(Internalizing)].
UMB Comprehensive Demographics Questionnaire
(Suyemoto et al., 2016)
The UMB Comprehensive Demographics Questionnaire was
used to gather comprehensive sociodemographic data includ-
ing detailed racial/ethnic identification, education, and
employment.
Procedures
All study procedures were approved by the Institutional
Review Board of the University of Southern Mississippi
(Protocol No.: CH18080202). Participants were recruited
both from the general student body (46.9%; compensated
with $50 gift cards) and from the psychology department
MMPI-3 IN BLACK AND WHITE AMERICAN UNDERGRADUATES 5
research pool (53.1%; compensated in course credits) as part
of a larger data collection effort. Participants from November
2018-March 2020 (n = 163, 64.2%) participated in-person in
the research laboratory. In-person participants completed
three blocks of study tasks counterbalanced to control for
participant fatigue. MMPI-2-RF-EX (via Pearson’s Q-Local)
and ASR (via ASEBA-WEB) were administered in the same
block. Due to COVID-19, participants after March 2020
(n = 91, 35.8%) participated virtually with permission and
guidance from the test developer and publisher.4 Demographic
information and the MMPI-2-RF-EX were administered first,
the latter following strict monitoring protocols for
tele-administration of the instrument including audio and
video monitoring (Corey & Ben-Porath, 2020; Joint Task
Force for the Development of Telepsychology Guidelines for
Psychologists, 2013; NCS Pearson, 2021; University of
Minnesota Press, 2020). Other measures, including the ASR,
were completed online and independently within 24 h. The
data that support the findings of this study are available
from the corresponding author upon reasonable request.
Data analytic plan
Internal psychometric properties
MMPI-3 Substantive Scale T-score means (unrounded,
untruncated) were calculated for each racialized group.
Means were compared via t-tests, with Cohen’s d used as
the indicator of effect size (.20 = small, .50 = medium,
.80 = large; Cohen, 1988; Ellis, 2010). A T-score mean dif-
ference was considered statistically significant if the 95%
confidence interval around Cohen’s d did not include zero.
We considered a mean difference clinically meaningful if it
was ≥5 T-score points (Graham et al., 2022).
Convergent and discriminant validity
To address the second study aim, zero-order correlations
between MMPI-3 Substantive Scales (raw scores) and ASR
Scales (raw scores) were calculated. Validity coefficients were
interpreted following established benchmarks: 0.05-0.09 = very
small, 0.10-0.19 = small, 0.20-0.29 = medium, 0.30-0.39 = large,
and ≥0.40 = very large (Funder & Ozer, 2019). Correlations
were only considered to be meaningful if they reached a
medium effect size (r ≥ |.20|; Funder & Ozer, 2019), as
common method variance artificially strengthens correlation
coefficients to a degree that cannot be quantified.
4
Proportions of Black and White participants did not differ across
in-person vs. virtual participation, Χ2 (1) = 0.69, p = .408. Virtual partic-
ipants were slightly older than in-person [virtual = 22.70 (6.20), in-person
= 20.64 (5.16)], t(160.12) = -2.70, p=.008. Proportionally more males
(40.7% virtual, 16.1% in-person), and proportionally fewer females (82.6%
in-person, 57.1% virtual), participated virtually vs. in-person, Χ 2 (3) =
19.44, p<.001. Raw score mean differences between these groups were
observed for three of the 32 examined MMPI-3 scales: Malaise [MLS;
in-person = 3.04(1.95), virtual = 3.62(1.77)], t(252) = -2.31, p = .021; Ideas
of Persecution [RC6; in-person = 2.55 (2.67), virtual = 1.90(2.10)], t(225.26)
= 2.12, p = .035; and Substance Abuse [SUB; in-person = 1.19(1.46),
virtual = 1.76(2.12)], t(138.80) = -2.27, p = .025. Amongst the 13 ASR
scales examined, mean differences were observed for only the Thought
Problems scale [(in-person = 5.10(3.20), virtual = 3.28(2.91)], t(250) =
4.46, p<.001.
6 ANESTIS ET AL.

Table 2. MMPI-3 T-score means and standard deviations in Black and White American undergraduate students.
White Black
Mean SD ω/α Mean SD ω/α T-Score Mean Difference Effect Size (d) Mean Comparison
Higher-Order
(H-O)
EID 56.16 11.47 .92/.92 54.25 10.11 .90/.88 1.92 0.17
THD 50.80 10.74 .80/.79 53.41 9.70 .71.71 −2.62 −0.25
BXD 47.87 8.77 .80/.81 47.22 7.41 .73/.75 0.65 0.08
Restructured
Clinical (RC)
RCd 54.98 11.46 .91/.91 54.25 9.35 .86/.86 0.73 0.07
RC1 54.89 12.12 .83/.84 54.07 10.87 .79/.80 0.81 0.07
RC2 52.23 10.79 .76/.76 49.65 10.32 .73/.75 2.58 0.24
RC4 46.51 7.95 .71/.72 45.28 6.15 .70/.49 1.23 0.17
RC6 51.94 10.80 .83/.82 54.26 9.09 .69/.70 −2.32 −0.23
RC7 58.67 12.58 .86/.86 58.59 12.49 .87/.87 0.07 0.01
RC8 53.85 12.90 .79/.78 56.12 11.38 .71/.70 −2.27 −0.18
RC9 53.77 11.52 .76/.77 53.54 12.19 .78/.79 0.24 0.02
Specific
Problems (SP)
Somatic/
Cognitive
MLS 50.84 10.81 .69/.69 50.15 10.46 .68/.70 0.69 0.07
NUC 53.33 11.70 .67/.67 54.69 10.52 .47/.59 −1.27 −0.11
EAT 52.22 11.80 .60/.62 51.35 10.68 .43/.52 0.87 0.08
COG 56.97 11.70 .88/.88 56.31 10.32 .85/.84 0.66 0.06
Internalizing
SUI 52.23 11.37 .68/.67 49.66 8.67 .68/.43 2.57 0.25
HLP 49.98 10.39 .67/.68 46.71 8.35 .50/.53 3.27 0.34*
SFD 55.58 12.14 .86/.85 52.82 11.33 .84/.84 2.77 0.23
NFC 55.74 11.09 .80/.80 55.87 10.92 .78/.78 −0.14 −0.01
STR 56.52 10.46 .63/.44 54.07 9.60 .56/.38 2.46 0.24
WRY 55.81 10.63 .80/.80 55.14 11.22 .85/.85 0.68 0.06
CMP 57.68 12.39 .80/.80 56.23 11.46 .74/.75 1.45 0.12
ARX 58.62 13.60 .88/.88 55.36 11.19 .82/.83 3.25 0.25
ANP 53.52 11.64 .89/.89 54.99 10.94 .87/.87 −1.47 −0.13
BRF 56.19 13.68 .55/.56 59.36 14.05 .50/.53 −3.17 −0.23
Externalizing
FML 52.87 11.29 .77/.77 52.92 10.12 .69/.69 −0.05 −0.00
JCP 45.92 7.71 .58/.60 46.97 6.46 .61/.37 −0.05 −0.01
SUB 47.14 9.68 .81/.80 47.56 6.79 .70/.51 −0.41 −0.05
IMP 51.88 11.98 .80/.79 49.92 10.82 .77/.76 1.96 0.17
ACT 54.89 11.24 .71/.71 55.18 10.47 .61/.63 −0.29 −0.03
AGG 49.62 10.70 .65/.63 52.36 9.98 .50/.52 −2.74 −0.26*
CYN 51.47 9.32 .74/.75 53.17 9.54 .69/.71 −1.71 −0.18
Interpersonal
SFI 46.86 10.61 .85/.84 50.39 9.09 .75/.76 −3.52 −0.35*
DOM 50.71 10.74 .74/.74 52.17 9.46 .52/.56 −1.47 −0.14
DSF 50.26 9.71 .74/.76 54.48 10.74 .77/.78 −4.22 −0.42*
SAV 52.48 12.27 .86/.86 51.31 11.07 .80/.81 1.17 0.10
SHY 55.35 11.66 .79/.79 54.52 11.00 .76/.76 0.83 0.07
Note. SD = standard deviation. ω = omega. α = alpha.
*Confidence interval does not contain zero.
EID (Emotional/Internalizing Dysfunction). THD (Thought Dysfunction). BXD (Behavioral/Externalizing Dysfunction). RCd (Demoralization). RC1 (Somatic Complaints).
RC2 (Low Positive Emotions). RC4 (Antisocial Behavior). RC6 (Ideas of Persecution). RC7 (Dysfunctional Negative Emotions). RC8 (Aberrant Experiences). RC9
(Hypomanic Activation). MLS (Malaise). NUC (Neurological Complaints). EAT (Eating Concerns). COG (Cognitive Complaints). SUI (Suicidal/Death Ideation). HLP
(Helplessness/Hopelessness). SFD (Self-Doubt). NFC (Inefficacy). STR (Stress). WRY (Worry). CMP (Compulsivity). ARX (Anxiety-Related Experiences). ANP (Anger
Proneness). BRF (Behavior-Restricting Fears). FML (Family Problems). JCP (Juvenile Conduct Problems). SUB (Substance Abuse). IMP (Impulsivity). ACT (Activation).
AGG (Aggression). CYN (Cynicism). SFI (Self-Importance). DOM (Dominance). DSF (Disaffiliativeness). SAV (Social Avoidance). SHY (Shyness).

Median convergent and discriminant validity coefficients
for hypothesized (convergent) and non-hypothesized (dis-
criminant) scale associations were estimated, similar to the
methods employed by Brown and Sellbom (2021) and
Handel et al. (2011). To test the hypothesis that convergent
associations would be stronger than divergent associations,
test of dependent correlations with Zou’s confidence interval
[using cocor (www.comparingcorrelations.org); Diedenhofen
& Musch, 2015; Zou, 2007] were implemented in the total
sample to examine differences in median correlation coef-
ficient magnitudes. Median convergent and discriminant
coefficients were considered significantly different from each
other in magnitude if Zou’s confidence interval did not
contain zero. To examine the consistency of convergent/
discriminant associations across Black and White partici-
pants, test of independent correlations with Zou’s confidence
interval [also using cocor; Diedenhofen & Musch, 2015;
Zou, 2007] were implemented to examine differences in
median correlation coefficient magnitudes between Black
and White participants. Again, coefficients were considered
significantly different in magnitude if Zou’s confidence inter-
val did not contain zero.

Results
Internal psychometric properties
Internal psychometric properties for MMPI-3 scales are
reported in Table 2. Overall, consistent with hypotheses,
individuals who identified as Black or White produced sim-
ilar T-scores on all scales. Small-sized statistically significant
mean differences were observed for four of 37 comparisons
(10.8% of comparisons). White participants produced higher
scores on Helplessness/Hopelessness [HLP; d = 0.34
(0.08,0.59)], whereas Black participants produced higher
scores on Disaffiliativeness [DSF; d = −0.42 (-0.68, −0.16)],
Self-Importance (SFI; d =-0.35 (-0.61, −0.09)], and Aggression
[AGG; d = −0.26 (-0.52, −0.01)]. However, no T-score means
differed by > 5 points, which was the a priori benchmark
set for clinical meaningfulness (Graham et al., 2022). The
median T-score difference between groups was quite small
[median = |1.45|; range = |0.05| [Family Problems (FML)
and Juvenile Conduct Problems (JCP)] - |4.22| (DSF)]. The
percentage of participants who exceeded the clinical T-score
cutoff for each examined scale is reported in Supplemental
Table S5 for descriptive purposes.
Convergent and discriminant validity
Zero-order correlations in the full sample are shown in
Table 3 and separately for Black and White participants in
Tables 4–6.
ASR internalizing problems and syndrome scales
Correlation coefficients for the associations between MMPI-3
Substantive Scales and ASR higher-order Internalizing and
lower-order Syndrome scales are located in Tables 3 (total
sample) and 4 (Black and White groups). All hypothesized
MMPI-3 scales [EID, Demoralization (RCd), Low Positive
Emotions (RC2), Dysfunctional Negative Emotions (RC7),
Suicidal/Death Ideation (SUI), HLP, Self-Doubt (SFD),
Inefficacy (NFC), STR, Worry (WRY), Compulsivity (CMP),
Anxiety-Related Experiences (ARX), Anger Problems (ANP),
and BRF] were meaningfully correlated with the higher-order
ASR Internalizing Problems scale in the total sample and
in both Black and White participants. Additionally, numer-
ous discriminant validity coefficients were considered mean-
ingful (78.3% of non-hypothesized rs ≥ |0.20|). Median
convergent validity coefficients (i.e., median coefficients of
hypothesized scales) were very large-sized in the total sample
(r = 0.58) and in both groups (White, r = 0.62; Black, r = 0.50).
Median discriminant coefficients (i.e., median coefficients
of non-hypothesized scales) were large-sized and above the
a priori-selected threshold for meaningful correlations (total,
r = 0.34; White, r = 0.38; Black, r = 0.30). Median convergent
associations indicated that the set of hypothesized MMPI-3
scales were substantially more correlated with ASR
Internalizing Problems than the set of non-hypothesized
scales (95% CI = 0.12, 0.36). Further, median convergent
and discriminant validity coefficients did not significantly
differ in magnitude between Black and White participants
MMPI-3 IN BLACK AND WHITE AMERICAN UNDERGRADUATES 7
(convergent 95% CI = −0.31, 0.06; discriminant 95%
CI = −0.31, 0.15).
A similar pattern was observed for MMPI-3 associations
with the lower-order ASR internalizing syndrome scales. All
hypothesized MMPI-3 scales (EID, RCd, RC2, RC7, SUI,
HLP, SFD, NFC, STR, WRY, CMP, ARX, ANP, and BRF)
were meaningfully correlated with the ASR Anxious-Depressed
syndrome scale in the total and in both White and Black
participants. Numerous non-hypothesized associations were
also considered meaningful (69.6% non-hypothesized rs ≥
|0.20|). Median convergent validity coefficients were very
large-sized (total, r = 0.60; White, r = 0.65; Black, r = 0.49),
while median discriminant coefficients were medium-large-
sized (total, r = 0.28; White, r = 0.35; Black, r = 0.23). Median
convergent associations indicated that the set of hypothe-
sized MMPI-3 scales were substantially more correlated with
ASR Anxious-Depressed than the set of non-hypothesized
scales (95% CI = 0.20, 0.45). As expected, median conver-
gent and discriminant validity coefficients did not signifi-
cantly differ in magnitude between Black and White
participants (convergent 95% CI = −0.35, 0.01; discriminant
95% CI = −0.36, 0.12).
The ASR Withdrawn syndrome scale was meaningfully
correlated with all hypothesized MMPI-3 scales [EID, RCd,
RC2, RC7, SUI, HLP, SFD, NFC, Social Avoidance (SAV),
Shyness (SHY), DSF] in the total sample and in Black and
White participants separately. Again, meaningful associations
were observed with numerous non-hypothesized MMPI-3
scales (76.9% non-hypothesized rs ≥ |0.20|). Median con-
vergent validity coefficients were very large-sized (total,
r = 0.49; White, r = 0.53; Black, r = 0.43), whereas median
discriminant coefficients were medium-large-sized (total,
r = 0.30; White, r = 0.31; Black, r = 0.28). As a set, the hypoth-
esized MMPI-3 scales were significantly more correlated
with ASR Withdrawn than the set of non-hypothesized
MMPI-3 scales (95% CI = 0.06, 0.32). Consistent with
expectations, median convergent and discriminant validity
coefficients did not significantly differ in magnitude between
Black and White participants (convergent 95% CI = −0.31,
0.10; discriminant 95% CI = −0.27, 0.20).
All hypothesized MMPI-3 scales [Somatic Complaints
(RC1), Neurological Complaints (NUC), MLS, Cognitive
Complaints (COG), and Eating Concerns (EAT)] were mean-
ingfully correlated with the ASR Somatic Complaints syn-
drome scale in the total sample and in both White and
Black participants. Several non-hypothesized associations
were also considered meaningful (84.4% non-hypothesized
rs ≥ |0.20|). Median convergent validity coefficients were
very large-sized in both groups (total, r = 0.48; White,
r = 0.50; Black, r = 0.45), while median discriminant coeffi-
cients were medium-large-sized (total, r = 0.30; White,
r = 0.32; Black, r = 0.29). As expected, the group of hypoth-
esized MMPI-3 scales were significantly more correlated
with ASR Somatic Complaints than the group of
non-hypothesizes scales (95% CI = 0.05, 0.31). Further,
median convergent and discriminant validity coefficients did
not significantly differ in magnitude between Black and
White participants (convergent 95% CI = −0.26, 0.15; dis-
criminant 95% CI = −0.27, 0.20).
8 ANESTIS ET AL.

Table 3. Convergent and discriminant validity coefficients amongst MMPI-3 scales and ASR scales in the total combined sample.
ASR ASR ASR ASR ASR
ASR ASR ASR Somatic ASR Aggressive ASR Rule ASR Thought Attention Mean
Internal-izing Anxious-Depressed Withdrawn Complaints External-izing Behavior Breaking intrusive Problems Problems Sub. Use
Mdn Conv 0.58 0.60 0.49 0.48 0.49 0.43 0.48 0.24 0.50 0.30 0.39
Mdn Disk 0.34 0.28 0.30 0.30 0.37 0.39 0.25 0.14 0.36 0.37 0.06

95% CI 0.12, 0.20, 0.06, 0.05, −0.01, −0.09, 0.09, −0.05, 0.02, −0.21, 0.18,
0.36* 0.45* 0.32* 0.31* 0.25 0.17 0.37* 0.25 0.26* 0.07 0.47*
H-O
EID .80† .81† .63† .53 .49 .59 .31 .14 .49 .60 .05
THD .40 .35 .33 .36 .44 .45 .34 .19 .50 .23 .07
BXD .24 .20 .19 .22 .58 .43 .68† .27 .30 .32 .41
RC
RCd .79 .81† .58 .55 .54 .60 .34 .23 .50 .63 .06
RC1 .62 .52 .39 .70† .46 .47 .38 .17 .54 .50 .15
RC2 .54 .54 .53 .29 .20 .28 .14 −0.05 .25 .44 .01
RC4 .23 .19 .19 .23 .49 .35 .63 .18 .27 .25 .38
RC6 .36 .34 .31 .27 .40 .39 .30 .23 .44 .20 .07
RC7 .64 .62 .49 .51 .52 .58 .33 .22 .49 .51 .03
RC8 .52 .45 .41 .50 .53 .52 .44 .23 .57† .38 .10
RC9 .20 .18 .14 .19 .52 .42 .48 .36 .34 .28 .15
S-P
Som./Cog.
MLS .56 .53 .38 .48 .41 .42 .34 .13 .37 .58 .17
NUC .47 .39 .38 .48 .41 .40 .39 .11 .43 .42 .18
EAT .44 .40 .25 .44 .30 .28 .30 .09 .37 .32 .21
COG .57 .53 .43 .47 .53 .52 .44 .24 .45 .71† .17
Internalizing
SUI .53 .53 .38 .38 .34 .37 .23 .12 .45 .32 .12
HLP .50 .49 .42 .33 .26 .32 .16 .07 .35 .36 −0.04
SFD .73 .77 .49 .49 .42 .46 .25 .23 .46 .54 .03
NFC .59 .60 .46 .38 .33 .38 .20 .15 .36 .61 .00
STR .57 .60 .32 .41 .30 .40 .12 .09 .32 .42 .06
WRY .61 .61 .40 .46 .42 .48 .23 .23 .46 .44 .07
CMP .34 .30 .30 .27 .28 .34 .11 .15 .30 .19 −0.05
ARX .73 .73 .46 .61 .50 .53 .33 .25 .56 .49 .12
ANP .38 .35 .34 .31 .60† .67† .36 .28 .37 .29 .05
BRF .40 .40 .24 .34 .24 .27 .17 .09 .34 .39 .00
Externalizing
FML .44 .40 .39 .36 .42 .46 .33 .11 .43 .37 .05
JCP .15 .10 .20 .12 .40 .30 .45 .20 .25 .20 .16
SUB .09 .08 .03 .12 .32 .16 .55 .06 .05 .15 .58†
IMP .30 .27 .23 .25 .53 .41 .59 .24 .36 .41 .26
ACT .19 .15 .11 .23 .38 .35 .26 .31† .30 .18 .00
AGG .26 .23 .28 .20 .58 .55 .48 .29 .36 .33 .14
CYN .34 .30 .33 .28 .41 .44 .33 .13 .37 .28 .09
Interpersonal
SFI −0.39 −0.42 −0.28 −0.22 −0.10 −0.13 −0.06 .00 −0.10 −0.35 −0.10
DOM −0.15 −0.19 −0.13 −0.02 .17 .15 .11 .16 .03 −0.14 −0.01
DSF .34 .24 .54 .21 .23 .38 .10 −0.09 .29 .22 −0.04
SAV .34 .28 .46 .18 .11 .24 −0.01 −0.12 .13 .22 −0.16
SHY .49 .48 .48 .30 .20 .27 .16 −0.04 .28 .38 −0.02
*Confidence interval does not contain zero. Bold = hypothesized convergent association. † = the single largest-sized coefficient in each subsample for each ASR
scale.
H-O (Higher-Order Scales). EID (Emotional/Internalizing Dysfunction). THD (Thought Dysfunction). BXD (Behavioral/Externalizing Dysfunction). RC (Restructured
Clinical Scales). RCd (Demoralization). RC1 (Somatic Complaints). RC2 (Low Positive Emotions). RC4 (Antisocial Behavior). RC6 (Ideas of Persecution). RC7
(Dysfunctional Negative Emotions). RC8 (Aberrant Experiences). RC9 (Hypomanic Activation). SP (Specific Problems Scales). MLS (Malaise). NUC (Neurological
Complaints). EAT (Eating Concerns). COG (Cognitive Complaints). SUI (Suicidal/Death Ideation). HLP (Helplessness/Hopelessness). SFD (Self-Doubt). NFC (Inefficacy).
STR (Stress). WRY (Worry). CMP (Compulsivity). ARX (Anxiety-Related Experiences). ANP (Anger Proneness). BRF (Behavior-Restricting Fears). FML (Family
Problems). JCP (Juvenile Conduct Problems). SUB (Substance Abuse). IMP (Impulsivity). ACT (Activation). AGG (Aggression). CYN (Cynicism). SFI (Self-Importance).
DOM (Dominance). DSF (Disaffiliativeness). SAV (Social Avoidance). SHY (Shyness).

ASR externalizing problems and syndrome scales
Correlation coefficients for the associations between MMPI-3
Substantive Scales and ASR higher-order Externalizing and
lower-order Syndrome scales are reported in Tables 3 (total
sample) and 5 (Black and White groups). All hypothesized
MMPI-3 scales [ANP, Behavioral/Externalizing Dysfunction
(BXD), Antisocial Behavior (RC4), Hypomanic Activation
(RC9), FML, JCP, Substance Abuse (SUB), AGG, Cynicism
(CYN), Impulsivity (IMP), and Activation (ACT)] were
meaningfully correlated with the higher-order ASR
Externalizing Problems scale in the total sample and in both
White and Black participants. Numerous discriminant valid-
ity coefficients were considered meaningful (88.5% of
non-hypothesized rs ≥ |0.20|). Median convergent validity
coefficients were very large-sized (total, r = 0.48; White,
r = 0.51; Black, r = 0.44), and median discriminant coefficients
were large-sized and above the a priori-selected threshold
for meaningful correlations (total, r = 0.37; White, r = 0.39;
Black, r = 0.29). Contrary to expectation, median convergent
 MMPI-3 IN BLACK AND WHITE AMERICAN UNDERGRADUATES 9

Table 4. Convergent and discriminant validity coefficients amongst MMPI-3 scales and ASR Internalizing domain scales.
ASR Internalizing ASR Anxious-Depressed ASR Withdrawn ASR Somatic Complaints
Between- Between- Between- Between-
Black r White r group CIs Black r White r group CIs Black r White r group CIs Black r White r group CIs
Mdn Conv 0.50 0.62 −0.31, 0.06 0.49 0.65 −0.35, 0.01 0.43 0.53 −0.31, 0.10 0.45 0.50 −0.26, 0.15
Mdn Disk 0.30 0.38 −0.31, 0.15 0.23 0.35 −0.36, 0.12 0.28 0.31 −0.27, 0.20 0.29 0.32 −0.27, 0.20
H-O
EID 0.70 0.84† 0.70 0.86† 0.52† 0.68† 0.51 0.54
THD 0.38 0.43 0.34 0.38 0.31 0.34 0.33 0.39
BXD 0.33 0.19 0.23 0.19 0.29 0.15 0.38 0.15
RC
RCd 0.72† 0.82 0.72† 0.85 0.49 0.63 0.55† 0.55
RC1 0.43 0.71 0.32 0.61 0.27 0.45 0.54 0.77†
RC2 0.44 0.57 0.45 0.57 0.44 0.59 0.23 0.31
RC4 0.29 0.20 0.16 0.19 0.27 0.17 0.40 0.15
RC6 0.30 0.40 0.30 0.37 0.27 0.32 0.18 0.33
RC7 0.56 0.69 0.51 0.69 0.43 0.52 0.48 0.53
RC8 0.42 0.59 0.31 0.53 0.31 0.45 0.48 0.53
RC9 0.23 0.19 0.21 0.17 0.15 0.13 0.21 0.18
S-P
Som/Cog.
MLS 0.43 0.62 0.37 0.60 0.28 0.43 0.45 0.50
NUC 0.21 0.61 0.11 0.54 0.25 0.45 0.26 0.60
EAT 0.54 0.37 0.48 0.36 0.31 0.23 0.55† 0.39
COG 0.48 0.61 0.44 0.58 0.34 0.48 0.42 0.50
Internalizing
SUI 0.46 0.55 0.46 0.55 0.24 0.46 0.42 0.36
HLP 0.39 0.53 0.43 0.50 0.36 0.47 0.14 0.39
SFD 0.65 0.76 0.70 0.80 0.45 0.53 0.39 0.53
NFC 0.55 0.61 0.54 0.64 0.38 0.51 0.41 0.37
STR 0.44 0.62 0.47 0.66 0.19 0.41 0.35 0.43
WRY 0.53 0.65 0.52 0.67 0.38 0.42 0.40 0.50
CMP 0.33 0.34 0.28 0.31 0.30 0.31 0.27 0.27
ARX 0.64 0.77 0.64 0.77 0.41 0.49 0.52 0.64
ANP 0.41 0.39 0.34 0.37 0.35 0.33 0.38 0.30
BRF 0.31 0.47 0.34 0.46 0.15 0.29 0.25 0.42
Externalizing
FML 0.36 0.48 0.34 0.42 0.33 0.43 0.25 0.42
JCP 0.18 0.13 0.05 0.12 0.24 0.18 0.28 0.05
SUB 0.21 0.05 0.14 0.05 0.16 −0.02 0.27 0.06
IMP 0.37 0.25 0.33 0.24 0.35 0.17 0.29 0.22
ACT 0.15 0.21 0.12 0.17 0.03 0.16 0.21 0.24
AGG 0.26 0.29 0.20 0.27 0.21 0.31 0.29 0.18
CYN 0.30 0.38 0.24 0.35 0.30 0.34 0.26 0.30
Interpersonal
SFI −0.37 −0.38 −0.39 −0.42 −0.38 −0.27 −0.14 −0.23
DOM −0.16 −0.13 −0.21 −0.16 −0.15 −0.13 0.00 −0.10
DSF 0.22 0.45 0.12 0.37 0.47 0.59 0.10 0.32
SAV 0.24 0.38 0.17 0.33 0.39 0.50 0.15 0.19
SHY 0.38 0.55 0.33 0.55 0.36 0.54 0.29 0.31
Bold = hypothesized convergent association. † = the single largest-sized coefficient in each subsample for each ASR scale.
H-O (Higher-Order Scales). EID (Emotional/Internalizing Dysfunction). THD (Thought Dysfunction). BXD (Behavioral/Externalizing Dysfunction). RC (Restructured
Clinical Scales). RCd (Demoralization). RC1 (Somatic Complaints). RC2 (Low Positive Emotions). RC4 (Antisocial Behavior). RC6 (Ideas of Persecution). RC7
(Dysfunctional Negative Emotions). RC8 (Aberrant Experiences). RC9 (Hypomanic Activation). SP (Specific Problems Scales). MLS (Malaise). NUC (Neurological
Complaints). EAT (Eating Concerns). COG (Cognitive Complaints). SUI (Suicidal/Death Ideation). HLP (Helplessness/Hopelessness). SFD (Self-Doubt). NFC
(Inefficacy). STR (Stress). WRY (Worry). CMP (Compulsivity). ARX (Anxiety-Related Experiences). ANP (Anger Proneness). BRF (Behavior-Restricting Fears). FML
(Family Problems). JCP (Juvenile Conduct Problems). SUB (Substance Abuse). IMP (Impulsivity). ACT (Activation). AGG (Aggression). CYN (Cynicism). SFI
(Self-Importance). DOM (Dominance). DSF (Disaffiliativeness). SAV (Social Avoidance). SHY (Shyness).

associations indicated that the set of hypothesized MMPI-3
scales were not significantly more correlated with ASR
Externalizing Problems than the set of non-hypothesized
scales (95% CI = −0.01, 0.25). As hypothesized, median con-
vergent and discriminant validity coefficients did not sig-
nificantly differ in magnitude between Black and White
participants (convergent 95% CI = −0.28, 0.13; discriminant
95% CI = −0.34, 0.13).
Regarding the ASR Aggressive Behavior syndrome scale,
meaningful correlations were observed for all hypothesized
MMPI-3 scales (ANP, BXD, RC4, RC9, FML, JCP, AGG,
CYN, IMP, ACT) in the total sample and amongst White
participants and for all except the JCP scale amongst Black
participants (r = 0.16; r = 0.37 in the White sample). Notably,
the largest magnitude convergent correlation in the total
sample and in both groups was with the ANP scale (all
rs = 0.67), as opposed to AGG (total, r = 0.55; White, r = 0.57;
Black, r = 0.54), which directly assesses engaging in aggres-
sive behavior. Numerous non-hypothesized associations were
considered meaningful (88.9% of non-hypothesized rs ≥
|0.20|). Median convergent validity coefficients were very
large-sized in both groups (total, r = 0.43; White, r = 0.43;
Black, r = 0.46), and median discriminant coefficients were
large-very large-sized (total, r = 0.39; White, r = 0.44; Black,
r = 0.32). Inconsistent with expectations, the hypothesized
MMPI-3 scales were not significantly more correlated with
10 ANESTIS ET AL.
ASR Aggressive Behavior than the set of non-hypothesized
MMPI-3 scales (95% CI = −0.09, 0.17). As expected, median
convergent and discriminant validity coefficients did not
significantly differ in magnitude between Black and White
participants (convergent 95% CI = −0.20, 0.22; discriminant
95% CI = −0.35, 0.10).
The ASR Rule Breaking syndrome scale was meaningfully
correlated with all hypothesized MMPI-3 scales (BXD, RC4,
RC9, FML, JCP, SUB, AGG, CYN, IMP, ACT) in the total
sample and among White participants and for all except the
FML scale amongst Black participants (r = 0.14; r = 0.41 in
the White sample). Several meaningful associations were
also observed with non-hypothesized MMPI-3 scales (63.0%
of non-hypothesized rs ≥ |0.20|). Median convergent validity
coefficients were very large-sized (total, r = 0.48; White,
r = 0.53 = 0; Black, r = 0.42), whereas median discriminant
coefficients were medium-sized (total, r = 0.25; White,
r = 0.27; Black, r = 0.24). As expected, the group of hypoth-
esized MMPI-3 scales were significantly more correlated
with ASR Rule Breaking than the group of non-hypothesizes
scales (95% CI = 0.09, 0.37). Consistent with hypotheses,
median convergent and discriminant validity coefficients did
not significantly differ in magnitude between Black and
White participants (convergent 95% CI = −0.19, 0.23; dis-
criminant 95% CI = −0.28, 0.21).
Convergent associations with the ASR Intrusive syndrome
scale were the weakest overall. We hypothesized seven
MMPI-3 scales to be most strongly associated with ASR
Intrusive [BXD, RC4, RC9, IMP, ACT, SFI, and Dominance
(DOM)]. In the total sample, only four of the hypothesized
MMPI-3 scales demonstrated meaningful associations (BXD,
RC9, IMP, and ACT). These four scales also demonstrated
meaningful associations amongst White and Black partici-
pants; additionally, RC4 was meaningfully associated in the
Black sample only and DOM in the White sample only.
Several non-hypothesized associations were considered
meaningful (33.3% of non-hypothesized rs ≥ |0.20|). Median
convergent validity coefficients were medium-sized (total,
r = 0.24; White and Black, r = 0.23), while median discrimi-
nant coefficients were small-sized (total, r = 0.14; White,
r = 0.15; Black, r = 0.11). Inconsistent with expectations, the
hypothesized MMPI-3 scales were not significantly more
correlated with ASR Intrusive than the non-hypothesized
MMPI-3 scale set (95% CI = −0.05, 0.25). As hypothesized,
median convergent and discriminant validity coefficients did
not significantly differ in magnitude between Black and
White participants (convergent 95% CI = −0.25, 0.24; dis-
criminant 95% CI = −0.30, 0.21).
ASR thought and attention problems syndrome and
mean substance use scales
Thought Problems, Attention Problems, and Substance Use
fall outside of the ASR Internalizing and Externalizing hier-
archical dimensions. Correlation coefficients for the associ-
ations between MMPI-3 Substantive Scales and ASR Thought
Problems, Attention Problems, and Mean Substance Use are
reported in Tables 3 (total sample) and 6 (Black and White
groups).
All hypothesized MMPI-3 scales [Thought Dysfunction
(THD), RC6, and Aberrant Experiences (RC8)] were mean-
ingfully correlated with the ASR Thought Problems syn-
drome scale in the total sample and in both White and
Black participants; however, it is notable that for Black
participants the single largest magnitude coefficient was for
a non-hypothesized scale assessing generalized anxiety,
panic, and reexperiencing (ARX, r = 0.60). Numerous dis-
criminant validity coefficients were considered meaningful
(88.2% of non-hypothesized rs ≥ |0.20|). Median convergent
validity coefficients were very large-sized (total, r = 0.50;
White, r = 0.53; Black, r = 0.48), and median discriminant
coefficients were large-sized (total, r = 0.36; White, r = 0.35;
Black, r = 0.34). As expected, the group of hypothesized
MMPI-3 scales were significantly more correlated with ASR
Thought Problems than the group of non-hypothesized
scales (95% CI = 0.02, 0.26). Further, median convergent
and discriminant validity coefficients did not significantly
differ in magnitude between Black and White participants
(convergent 95% CI = −0.25, 0.14; discriminant 95%
CI = −0.25, 0.21).
Regarding the ASR Attention Problems syndrome scale,
we hypothesized six MMPI-3 scales would demonstrate the
strongest associations (COG, BXD, RC4, RC9, IMP, ACT).
This was the case amongst the White participants. In the
total sample and amongst Black participants, meaningful
correlations were observed for all hypothesized MMPI-3
scales except ACT. In the total sample and in both groups,
the largest correlation magnitude was between ASR Attention
Problems and COG, the scale that most directly assesses
problems with attention and concentration (total, r = 0.71;
White, r = 0.72; Black, r = 0.69). Numerous discriminant
validity coefficients were considered meaningful (87.1% of
non-hypothesized rs ≥ |0.20|). Median convergent validity
coefficients were medium-sized in the sample of Black par-
ticipants (r = 0.24) and large-sized in the total sample
(r = 0.30) and the White group (r = 0.34); median discrimi-
nant coefficients were uniformly large-sized (total, r = 0.37;
White, r = 0.37; Black, r = 0.33). Contrary to expectation,
median convergent associations indicated that the set of
hypothesized MMPI-3 scales were not significantly more
correlated with ASR Attention Problems than the set of
non-hypothesized scales (95% CI = −0.21, 0.07). As hypoth-
esized, median convergent and discriminant validity coeffi-
cients did not significantly differ in magnitude between
Black and White participants (convergent 95% CI = −0.34,
0.14; discriminant 95% CI = −0.27, 0.18).
ASR Mean Substance Use was meaningfully correlated
with all hypothesized MMPI-3 scales (BXD, RC4, SUB, IMP)
in the total sample and amongst White and Black partici-
pants. In all, the largest correlation magnitude was between
ASR Mean Substance Use and SUB, the only MMPI-3 scale
that directly assesses past and current substance use (total,
r = 0.58; White, r = 0.61; Black, r = 0.54). In contrast to the
other ASR scales, only one meaningful association was
observed with non-hypothesized MMPI-3 scales (3.0% of
non-hypothesized rs ≥ |0.20|). Median convergent validity
coefficients were very large-sized in the sample of White
participants (r = 0.40) and large-sized in the total sample
 MMPI-3 IN BLACK AND WHITE AMERICAN UNDERGRADUATES 11

Table 5. Convergent and discriminant validity coefficients amongst MMPI-3 scales and ASR Externalizing domain scales.
ASR Externalizing ASR Aggressive Behavior ASR Rule Breaking ASR intrusive
Between- Between- Between Between-
Black r White r group CIs Black r White r group CIs Black r White r group CIs Black r White r group CIs
Mdn Conv 0.44 0.51 −0.28, 0.13 0.46 0.43 −0.20, 0.22 0.42 0.50 −0.19, 0.23 0.23 0.23 −0.25, 0.24
Mdn Disk 0.29 0.39 −0.34, 0.13 0.32 0.44 −0.35, 0.10 0.24 0.27 −0.28, 0.21 0.11 0.15 −0.30, 0.21
H-O
EID 0.46 0.50 0.56 0.61 0.26 0.33 0.10 0.13
THD 0.32 0.51 0.32 0.52 0.31 0.36 0.04 0.29
BXD 0.55 0.60† 0.46 0.43 0.55† 0.73† 0.28 0.26
RC
RCd 0.50 0.55 0.58 0.62 0.26 0.37 0.21 0.22
RC1 0.46 0.47 0.44 0.50 0.39 0.37 0.19 0.16
RC2 0.10 0.24 0.20 0.33 0.00 0.20 −0.09 −0.07
RC4 0.44 0.51 0.31 0.38 0.55† 0.66 0.21 0.15
RC6 0.23 0.49 0.18 0.50 0.27 0.32 0.10 0.30
RC7 0.51 0.52 0.55 0.60 0.34 0.32 0.23 0.22
RC8 0.47 0.57 0.42 0.57 0.47 0.44 0.19 0.28
RC9 0.49 0.54 0.40 0.43 0.44 0.50 0.34† 0.37†
S-P
Som/Cog.
NUC 0.25 0.49 0.23 0.49 0.28 0.44 0.03 0.15
MLS 0.38 0.42 0.39 0.44 0.29 0.36 0.13 0.12
EAT 0.29 0.30 0.31 0.27 0.28 0.30 −0.02 0.13
COG 0.45 0.57 0.47 0.56 0.32 0.49 0.19 0.26
Internalizing
SUI 0.23 0.37 0.25 0.43 0.14 0.26 0.10 0.11
HLP 0.19 0.29 0.22 0.38 0.15 0.16 0.02 0.05
SFD 0.35 0.45 0.39 0.51 0.20 0.27 0.17 0.23
NFC 0.38 0.31 0.40 0.37 0.22 0.19 0.24 0.12
STR 0.29 0.30 0.43 0.39 0.09 0.13 −0.04 0.12
WRY 0.45 0.41 0.50 0.47 0.24 0.22 0.24 0.22
CMP 0.28 0.28 0.33 0.34 0.13 0.10 0.10 0.17
ARX 0.52 0.49 0.50 0.56 0.38 0.31 0.31 0.21
ANP 0.62† 0.60† 0.67† 0.67† 0.43 0.33 0.22 0.34
BRF 0.14 0.31 0.14 0.34 0.16 0.19 −0.01 0.16
Externalizing
FML 0.38 0.44 0.49 0.44 0.14 0.41 0.12 0.10
JCP 0.32 0.43 0.16 0.37 0.40 0.47 0.30 0.15
SUB 0.22 0.36 0.17 0.16 0.33 0.62 −0.03 0.09
IMP 0.53 0.52 0.46 0.39 0.54 0.61 0.23 0.23
ACT 0.40 0.38 0.32 0.36 0.35 0.23 0.33 0.32
AGG 0.58 0.59 0.57 0.54 0.48 0.49 0.23 0.36
CYN 0.41 0.43 0.45 0.44 0.28 0.36 0.13 0.15
Interpersonal
SFI −0.13 −0.07 −0.21 −0.10 −0.01 −0.07 -0.01 0.07
DOM 0.14 0.20 0.15 0.16 0.09 0.12 0.04 0.23
DSF 0.11 0.34 0.27 0.47 0.00 0.18 −0.20 0.02
SAV 0.09 0.11 0.21 0.27 −0.01 −0.01 −0.12 −0.13
SHY 0.24 0.18 0.25 0.28 0.19 0.14 0.07 −0.10
Bold = hypothesized convergent association. † = the single largest-sized coefficient in each subsample for each ASR scale.
H-O (Higher-Order Scales). EID (Emotional/Internalizing Dysfunction). THD (Thought Dysfunction). BXD (Behavioral/Externalizing Dysfunction). RC (Restructured
Clinical Scales). RCd (Demoralization). RC1 (Somatic Complaints). RC2 (Low Positive Emotions). RC4 (Antisocial Behavior). RC6 (Ideas of Persecution). RC7
(Dysfunctional Negative Emotions). RC8 (Aberrant Experiences). RC9 (Hypomanic Activation). SP (Specific Problems Scales). MLS (Malaise). NUC (Neurological
Complaints). EAT (Eating Concerns). COG (Cognitive Complaints). SUI (Suicidal/Death Ideation). HLP (Helplessness/Hopelessness). SFD (Self-Doubt). NFC
(Inefficacy). STR (Stress). WRY (Worry). CMP (Compulsivity). ARX (Anxiety-Related Experiences). ANP (Anger Proneness). BRF (Behavior-Restricting Fears). FML
(Family Problems). JCP (Juvenile Conduct Problems). SUB (Substance Abuse). IMP (Impulsivity). ACT (Activation). AGG (Aggression). CYN (Cynicism). SFI
(Self-Importance). DOM (Dominance). DSF (Disaffiliativeness). SAV (Social Avoidance). SHY (Shyness).

(r = 0.39) and among the Black participants (r = 0.38), Discussion

whereas median discriminant coefficients were negligible in
the White sample (r = 0.02), very small in the total sample
(r = 0.06), and small in the Black sample (r = 0.10). Median
convergent associations indicated that the set of hypothe-
sized MMPI-3 scales were substantially more correlated with
ASR Mean Substance Use than the set of non-hypothesized
scales (95% CI = 0.18, 0.47). Further, median convergent
and discriminant validity coefficients did not significantly
differ in magnitude between Black and White participants
(convergent 95% CI = −0.25, 0.19; discriminant 95%
CI = −0.18, 0.33).
The current study used a sample of undergraduate students
to examine internal and external psychometric properties
of the MMPI-3 with a particular focus on the impact of
racialized group membership. Although a substantial strength
of the MMPI-3 test development process is the acquisition
of a new, contemporary normative sample representative of
the US population in terms of race, education, and age
(Ben-Porath & Tellegen, 2020a), confidence in the generality
of the instrument would be augmented by deliberate research
on internal and external validity properties of the MMPI-3
12 ANESTIS ET AL.

Table 6. Convergent and discriminant validity coefficients amongst MMPI-3 scales and ASR thought problems, attention problems, and mean substance use.
ASR Thought Problems ASR Attention Problems ASR Mean Substance Use
Between- Between- Between-
Black r White r group CIs Black r White r group CIs Black r White r group CIs
Mdn Conv 0.48 0.53 −0.25, 0.14 0.24 0.34 −0.34, 0.14 0.38 0.40 −0.25, 0.19
Mdn Disk 0.34 0.35 −0.25, 0.21 0.33 0.37 −0.27, 0.18 0.10 0.02 −0.18, 0.33
H-O
EID 0.49 0.49 0.61 0.60 0.10 0.02
THD 0.48 0.53 0.19 0.25 0.20 −0.01
BXD 0.41 0.26 0.25 0.35 0.40 0.41
RC
RCd 0.53 0.49 0.64 0.62 0.06 0.06
RC1 0.39 0.61† 0.43 0.53 0.19 0.12
RC2 0.19 0.26 0.34 0.48 0.07 −0.03
RC4 0.36 0.23 0.23 0.26 0.36 0.39
RC6 0.37 0.48 0.09 0.26 0.14 0.04
RC7 0.49 0.50 0.51 0.51 0.06 0.02
RC8 0.51 0.61† 0.33 0.41 0.22 0.05
RC9 0.34 0.34 0.22 0.32 0.26 0.09
S-P
Som/Cog.
MLS 0.30 0.40 0.47 0.63 0.26 0.11
NUC 0.22 0.54 0.19 0.54 0.26 0.15
EAT 0.48 0.32 0.35 0.31 0.15 0.24
COG 0.40 0.48 0.69† 0.72† 0.21 0.14
Internalizing
SUI 0.36 0.47 0.32 0.33 0.18 0.09
HLP 0.33 0.35 0.18 0.43 −0.02 −0.05
SFD 0.48 0.45 0.54 0.54 −0.03 0.05
NFC 0.34 0.36 0.63 0.60 0.09 −0.05
STR 0.25 0.35 0.44 0.41 0.17 0.00
WRY 0.45 0.46 0.45 0.44 0.03 0.10
CMP 0.25 0.32 0.25 0.16 −0.04 −0.05
ARX 0.60† 0.55 0.50 0.48 0.14 0.10
ANP 0.44 0.35 0.36 0.25 0.20 −0.04
BRF 0.24 0.41 0.43 0.37 −0.01 0.01
Externalizing
FML 0.46 0.42 0.31 0.40 0.07 0.04
JCP 0.29 0.24 0.14 0.23 0.06 0.21
SUB 0.14 0.01 0.17 0.14 0.54† 0.61†
IMP 0.42 0.32 0.34 0.45 0.30 0.24
ACT 0.24 0.33 0.14 0.20 0.16 −0.09
AGG 0.46 0.33 0.28 0.37 0.32 0.04
CYN 0.29 0.42 0.20 0.33 0.25 0.01
Interpersonal
SFI −0.16 −0.06 −0.41 −0.32 −0.05 −0.12
DOM −0.03 0.07 −0.26 −0.09 0.10 −0.05
DSF 0.14 0.41 0.20 0.26 0.02 −0.08
SAV 0.00 0.18 0.26 0.20 −0.09 −0.20
SHY 0.23 0.31 0.44 0.35 −0.04 0.00
Bold = hypothesized convergent association. † = the single largest-sized coefficient in each subsample for each ASR scale.
H-O (Higher-Order Scales). EID (Emotional/Internalizing Dysfunction). THD (Thought Dysfunction). BXD (Behavioral/Externalizing Dysfunction). RC (Restructured
Clinical Scales). RCd (Demoralization). RC1 (Somatic Complaints). RC2 (Low Positive Emotions). RC4 (Antisocial Behavior). RC6 (Ideas of Persecution). RC7
(Dysfunctional Negative Emotions). RC8 (Aberrant Experiences). RC9 (Hypomanic Activation). SP (Specific Problems Scales). MLS (Malaise). NUC (Neurological
Complaints). EAT (Eating Concerns). COG (Cognitive Complaints). SUI (Suicidal/Death Ideation). HLP (Helplessness/Hopelessness). SFD (Self-Doubt). NFC (Inefficacy).
STR (Stress). WRY (Worry). CMP (Compulsivity). ARX (Anxiety-Related Experiences). ANP (Anger Proneness). BRF (Behavior-Restricting Fears). FML (Family
Problems). JCP (Juvenile Conduct Problems). SUB (Substance Abuse). IMP (Impulsivity). ACT (Activation). AGG (Aggression). CYN (Cynicism). SFI (Self-Importance).
DOM (Dominance). DSF (Disaffiliativeness). SAV (Social Avoidance). SHY (Shyness).

across diverse population, contexts, and settings. Additionally,
the MMPI-3 retains the hierarchical-dimensional organiza-
tion and interpretive framework designed to align the instru-
ment with contemporary theory and knowledge on the
structure of psychopathology and personality (Ben-Porath
& Tellegen, 2020b, 2020a), yet limited research is available
on the actual alignment of the MMPI-3 hierarchical domains
with other hierarchical-dimensional tools commonly used
in clinical assessment. Results of the current study provide
preliminary support for assertions that 1) MMPI-3 substan-
tive scale mean T-scores are comparable between White and
Black American undergraduate students; 2) MMPI-3 scales
are associated with ASR scale scores in expected ways in
an undergraduate sample with regard to internalizing prob-
lems, rule breaking and impulsivity, thought problems, and
substance use; and 3) convergent and discriminant associ-
ations between MMPI-3 and ASR scales are consistent across
White and Black American undergraduate students.
Extant literature found racialized group membership did
not meaningfully impact test scores on earlier MMPI iter-
ations (Marek et al., 2015; Whitman et al., 2019). We, there-
fore, hypothesized that the MMPI-3 scales would exhibit
similar T-score means across Black and White undergrad-
uates. Our results were consistent with this expectation.
Four small-sized statistically significant mean differences
were observed (10.8% of comparisons; White participants

higher: HLP; Black participants higher: DSF, SFI, AGG);
however, no mean T-score differences reached the a priori
benchmark of clinical meaningfulness (>5 T-score points;
Graham et al., 2022). Such results suggest consistency in
responding and T-score conversion between White and Black
American undergraduate students.
Examination of convergent/discriminant associations with
the ASR was important and novel in two ways: 1) empirical
correlates (derived from convergent/discriminant associa-
tions) provide the evidentiary support for test interpretive
statements and thus warrant close examination in relation
to sociodemographic group membership and 2) limited
research exists on the associations of MMPI scales with
other hierarchical-dimensional tools commonly used in clin-
ical assessment (like the ASR). We expected hypothesized
convergent relationships to be substantially stronger than
non-hypothesized discriminant relationships. This hypothesis
was supported in analyses concerning the higher-order ASR
Internalizing Problems scale, the three ASR internalizing
syndrome scales (Anxious-Depressed, Withdrawn, Somatic
Complaints), ASR Rule Breaking, ASR Thought Problems,
and ASR Mean Substance Use. In each of these instances,
the median convergent correlation coefficient was signifi-
cantly larger than the median discriminant correlation coef-
ficient, suggesting that these sets of MMPI-3 scales and
their theoretical ASR counterparts are assessing similar
constructs.
Hypotheses about convergent and discriminant relation-
ships, however, were not supported for the higher-order
ASR Externalizing Problems scale and the Aggressive
Behavior, Intrusive, and Attention Problems syndrome scales.
With each of these scales, the median correlation coefficient
of hypothesized MMPI-3 scale sets was not significantly
larger than the median correlation coefficient of
non-hypothesized scale sets. One likely possibility for this
finding is differences between the instruments regarding the
extraction of demoralization. Demoralization represents per-
vasive and indistinct distress associated with multiple mental
and physical health conditions and is thought to contribute
to the high intercorrelations and poor discriminant validity
observed in many self-report measures of personality and
psychopathology (Tellegen et al., 2003, 2009; Uliaszek et al.,
2019). Poor discriminant validity was a critique of the
MMPI/MMPI-2 Clinical Scales, and work done by Tellegen
et al. (2003) to extract and isolate shared variance related
to demoralization to facilitate distinct, specific Restructured
Clinical Scales represents a major development in the assess-
ment literature (Tellegen et al., 2003). Despite the longstand-
ing concerns raised regarding demoralization (Tellegen,
1985), attention to this construct is generally not seen across
other assessment instruments, and indeed the ASR scales
exhibit sizable intercorrelations suggesting less specificity
across scales and likely overall demoralization saturation
(Achenbach & Rescorla, 2003; see also Supplemental Table
S4). Further, MMPI-3 RCd demonstrated very-large sized
correlations with ASR Externalizing Problems, Aggressive
Behavior, and Attention Problems in the total sample (rs
range from 0.54-0.63), lending credence to our supposition
that the ASR may be saturated with demoralization content.
MMPI-3 IN BLACK AND WHITE AMERICAN UNDERGRADUATES 13
Demoralization saturation in the ASR syndrome scales likely
contributed to inconsistencies observed in convergent and
discriminant validity evidence. Refinement of the ASEBA
measures to improve scale specificity as well replication of
the current study with other hierarchical-dimensional model
scales that are not statured with demoralization as criteria
seems warranted.
We expected to observe similar magnitudes of convergent/
discriminant associations with the ASR across Black and
White undergraduates. Consistent with this expectation,
between-group magnitude comparisons indicated that
median convergent and discriminant validity coefficients did
not significantly differ between White and Black Americans.
We thus tentatively conclude that interpretive statements
derived from these correlations are applicable to both White
and Black American undergraduate students. Ample litera-
ture outside the assessment domain demonstrates that the
lived experiences among individuals of diverse racial iden-
tities can lead to deleterious mental health impacts (Williams,
2018), but the current study is consistent with prior MMPI
research suggesting that mere membership in a racialized
group may not impact test scores. Our study represents an
important step in investigating the impact of racial group
membership on MMPI-3 scores. Furthermore, consideration
of race-based bias in the ASEBA literature seems to be
limited to the ASEBA Child Behavior Checklist (Achenbach
& Rescorla, 2000; Gross et al., 2006), an informant measure
of youth behavior, and we are unaware of literature that
examines potential race-based bias in the ASEBA adult
forms. Future work should examine predictive differential
validity in the ASEBA measures and should also examine
other ASEBA forms in relation to the MMPI-3 (e.g., infor-
mant measures).
Overall, there is a great need in psychological science
for intentional research that actively recruits individuals of
diverse racial backgrounds. The American Psychological
Association’s (2019) guidelines on race and ethnicity empha-
size the importance of increasing representation in psycho-
logical science by intentionally conducting research that
considers, recruits, and involves racially diverse individuals.
Additionally, professional practice guidelines for personality
assessment emphasize the importance of giving appropriate
attention to diversity which includes considerations of diver-
sity when choosing an assessment instrument (Krishnamurthy
et al., 2022). These guidelines highlight the need for psy-
chologists to be familiar with literature regarding how diver-
sity impacts personality assessment. Psychological assessment
researchers should work to, at a minimum, report analyses
by demographic groups to further these efforts and highlight
potential group-based differences, particularly with psycho-
logical assessments that are widely disseminated and utilized.
A notable strength of the current work is the use of a
standardized demographic assessment to gain a comprehen-
sive look at participants’ identities. Future work should con-
sider adopting comprehensive and standardized measures to
increase the consistency of definitions for demographic-based
groups across studies and thereby increase confidence in
and potential replicability of findings. Furthermore, research-
ers should be mindful of biases in psychological research
14 ANESTIS ET AL.
and the impact that their research may have on marginalized
populations. Future research should also include measures
of racialized experiences (e.g., perceived discrimination) to
determine whether these unique experiences influence test
scores and other aspects of the assessment process (see
Whatley et al., 2003, as an example).
An important limitation of the current study is that we
examined only slope bias (i.e., differential associations with
extra-test criteria), and we did not examine intercept bias
(i.e., systematic over or underprediction of relevant criteria).
This decision was made in light of two factors: 1) our sam-
ple was underpowered to test intercept bias via moderated
multiple regression, the accepted method for examining
intercept bias, and 2) extant literature indicates that regres-
sion “yields inaccurate inferences for prediction bias hypoth-
eses” (Culpepper, 2012, p. 561). To further elaborate on the
latter, conclusions about intercept bias may be erroneous
due to methodological and statistical artifacts inherent in
the moderated multiple regression approach (see Aguinis
et al., 2010, for a comprehensive review). Artifacts include
insufficient power to detect small effects [i.e., the median
effect size (f2) of categorical moderators (such as race group
membership) is .002; Aguinis et al., 2005]. Alternative meth-
ods for examining group differences in assessment respond-
ing that do not rely on moderated multiple regression, such
as those used in the current study and prior research (e.g.,
Anestis et al., 2019; Handel et al., 2011), are necessary to
increase confidence in assertions about test. Validity coef-
ficients play a critical role in shaping test interpretive state-
ments, and research such as the current study that closely
examines validity coefficients (i.e., slope bias) is necessary
but not sufficient alone to support the use of tests with
diverse populations. Tests of measurement invariance are
particularly important to ensure that tests are measuring
the same constructs in different groups (e.g., Han et al.,
2019). Such research with the MMPI-3 and the ASR will
be critical moving forward, in light of recent work suggest-
ing lack of construct equivalence across Black and White
Americans on other personality/psychopathology measures
(Bagby et al., 2022; Charles et al., 2022).
The current study’s used of a young undergraduate sam-
ple, and conclusions may be best considered in the context
of other college student samples such as the large college
validation sample described in the MMPI-3 Technical Manual
(Ben-Porath & Tellegen, 2020a). The student population of
the current study, however, features a larger proportion of
Black individuals (32.4%) relative to other college students
samples featured in MMPI research (e.g., Forbey et al., 2010;
Whitman & Ben-Porath, 2021). Additionally, a majority were
employed in addition to being enrolled at the university
(57.1%). As such, it is possible that the current sample may
be more representative of community members and more
applicable to clinical settings than other undergraduate
samples.
Consistent with general findings on the overall emotional
health of college students and the predominance of inter-
nalizing psychopathology in this population (Auerbach et al.,
2018), this sample reported relatively high levels of negative
emotionality-related psychopathology (e.g., roughly one third
of each subsample exceeded the clinical cutoff for RC7; see
Supplemental Table S5). Nevertheless, our sample exhibited
a somewhat restricted range of psychopathology scores (see
Table 2), and neither the Black nor White subsamples had
mean T-scores within the clinical range which must be
considered in terms of generalizability to samples with
higher base rates of psychopathology. There was notably
low endorsement of externalizing psychopathology in the
current sample. Restricted range of externalizing likely
impacted the validity coefficients examined, as results are
more consistent and stronger among the internalizing scales.
It will be important for future work to use diverse clinical
samples, particularly those with higher externalizing
psychopathology.
Other study limitations should be noted. We exclusively
used self-report measures, and validity coefficients may be
somewhat inflated due to shared method variance. Future
research would benefit from gathering data from
multi-method sources such as informants and behavioral or
objective indicators of psychopathology with diverse popu-
lations (e.g., Anestis et al., 2022). The sample was majority
female/women, and this was especially true within the Black
subsample (i.e., only 10 Black males). Our study focuses
exclusively on Black and White individuals in the United
States; future research should work to include other racial
and ethnic groups and explore intersectionality (e.g., with
socioeconomic status, gender) to expand confidence in the
utilization of psychological assessments with individuals tra-
ditionally underrepresented in research.
Despite the study’s limitations, this work contributes to
the evidence base supporting the use of the MMPI-3 with
individuals of diverse racialized group membership and
points to numerous directions for next steps in understand-
ing how MMPI-3 scale scores function in diverse popula-
tions. Current study findings suggest that MMPI-3 scale
interpretative statements can be applied similarly across
Black and White American undergraduate students, although
these conclusions should be considered preliminary at this
time. Additionally, findings provide a novel form of support
for the MMPI-3’s convergent and discriminant validity with
a similarly structured measure of psychopathology (i.e., the
ASEBA ASR). The current work is an important step in
diversifying psychological assessment research and empha-
sizes the importance of establishing an empirical basis for
the use of psychological assessments across many individ-
uals. Further research that broadens the psychological assess-
ment literature to consider many other aspects of diversity
is vital.
Authorship CRediT statement
Joye C. Anestis—Conceptualization, Data curation, Formal
analysis, Funding acquisition, Methodology, Project admin-
istration, Resources, Software, Supervision, Validation,
Writing—original draft, Writing—review and editing; Taylor
R. Rodriguez - Data curation, Investigation, Software,
Writing—original draft, Writing—review and editing; Chloe
O’Dell—Writing—original draft; Olivia M. Preston - Data

curation, Investigation, Software, Writing—review and edit-
ing; Tiffany M. Harrop - Data curation, Investigation,
Software, Writing—review and editing; Nora E. Charles -
Project administration, Writing—review and editing.
Acknowledgments
The authors would like to thank Jacob Finn and Tayla Lee for their
feedback both during the development stage of this project and on
earlier drafts of this manuscript. They are grateful to Chloe Woodling
for assistance with data management. They would also like to thank
the following undergraduate research assistants who were instrumental
in collecting these data: Brelynne Baldwin, Gabrielle Craig, Thomas
Hicks, and Mary Roth.
Funding
Joye C. Anestis received grant funding for this research project from
the University of Minnesota Press. The data that support the findings
of this study are available from the corresponding author upon rea-
sonable request.
ORCID
Joye C. Anestis http://orcid.org/0000-0002-0014-8502
Taylor R. Rodriguez http://orcid.org/0000-0002-7110-3993
Chloe O’Dell http://orcid.org/0000-0002-6536-1363
Olivia C. Preston http://orcid.org/0000-0003-1443-3125
Tiffany M. Harrop http://orcid.org/0000-0001-9630-2705
Nora E. Charles http://orcid.org/0000-0002-9638-4832
References
Achenbach, T. M., Ivanova, M. Y., & Rescorla, L. A. (2017). Empirically
based assessment and taxonomy of psychopathology for ages 1½–
90+ years: Developmental, multi-informant, and multicultural find-
ings. Comprehensive Psychiatry, 79, 4–18. https://doi.org/10.1016/j.
comppsych.2017.03.006
Achenbach, T. M., & Rescorla, L. A. (2000). Manual for the ASEBA
preschool forms and profiles. University of Vermont, Research center
for Children, Youth, & Families.
Achenbach, T. M., & Rescorla, L. A. (2003). Manual for the ASEBA
adult forms & profiles. University of Vermont, Research Center for
Children, Youth, & Families.
Aguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005). Effect
size and power in assessing moderating effects of categorical vari-
ables using multiple regression: A 30-year review. The Journal of
Ap p l i e d Psycholog y, 90(1), 94–107. https://doi.
org/10.1037/0021-9010.90.1.94
Aguinis, H., Culpepper, S. A., & Pierce, C. A. (2010). Revival of test
bias research in preemployment testing. The Journal of Applied
Psychology, 95(4), 648–680. https://doi.org/10.1037/a0018714
American Psychological Association, APA Task Force on Race and
Ethnicity Guidelines in Psychology. (2019). Race and ethnicity guide-
lines in psychology: Promoting responsiveness and equity. Retrieved
from http://www.apa.org/about/policy/race-and-ethnicity-in-psychol-
ogy.pdf
Anestis, J. C., Preston, O. C., Harrop, T. M., & Sellbom, M. (2019).
The intersection of sociodemographic characteristics within the
nomological network of the triarchic psychopathy model in a fo-
rensic sample. Journal of Criminal Justice, 61, 13–25. https://doi.
org/10.1016/j.jcrimjus.2019.02.001
Anestis, J. C., Preston, O. C., Rodriguez, T. R., & Harrop, T. M. (2022).
MMPI-3 scale predictors of reactions to ostracism in a sample of
MMPI-3 IN BLACK AND WHITE AMERICAN UNDERGRADUATES 15
racially diverse college students. Psychological Assessment, 34(6),
503–516. https://doi.org/10.1037/pas0001113
Arbisi, P. A., Ben-Porath, Y. S., & McNulty, J. (2002). A comparison
of MMPI-2 validity in African American and Caucasian psychiatric
inpatients. Psychological Assessment, 14(1), 3–15. https://doi.
org/10.1037/1040-3590.14.1.3
Auerbach, R. P., Mortier, P., Bruffaerts, R., Alonso, J., Benjet, C.,
Cuijpers, P., Demyttenaere, K., Ebert, D. D., Green, J. G., Hasking,
P., Murray, E., Nock, M. K., Pinder-Amaker, S., Sampson, N. A.,
Stein, D. J., Vilagut, G., Zaslavsky, A. M., & Kessler, R. C. (2018).
WHO World Mental Health Surveys International College Student
Project: Prevalence and distribution of mental disorders. Journal of
Abnormal Psychology, 127(7), 623–638. https://doi.org/10.1037/
abn0000362
Bagby, R. M., Keeley, J. W., Williams, C. C., Mortezaei, A., Ryder, A.
G., & Sellbom, M. (2022). Evaluating the measurement invariance
of the Personality Inventory for DSM-5 (PID-5) in Black Americans
and White Americans. Psychological Assessment, 34(1), 82–90. https://
doi.org/10.1037/pas0001085
Ben-Porath, Y. S., & Tellegen, A. (2020a). Minnesota Multiphasic
Personality Inventory-3 (MMPI-3): Technical manual. University of
Minnesota Press.
Ben-Porath, Y. S., & Tellegen, A. (2020b). MMPI-3 (Minnesota
Multiphasic Personality Inventory-3): Manual for administration, scor-
ing, and interpretation. University of Minnesota Press.
Brown, T. A., & Sellbom, M. (2021). Associations between MMPI-3
scale scores and the DSM-5 personality disorders. Journal of Clinical
Psychology, 77(12), 2943–2964. https://doi.org/10.1002/jclp.23230
Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., &
Kaemmer, B. (1989). Minnesota Multiphasic Personality Inventory-2
(MMPI-2): Manual for Administration and Scoring. University of
Minnesota Press.
Butcher, J. N., Williams, C. L., Graham, J. R., Archer, R. P., Tellegen,
A., Ben-Porath, Y. S., & Kaemmer, B. (1993). MMPI-A (Minnesota
Multiphasic Personality Inventory-Adolescent): Manual for adminis-
tration, scoring, and interpretation. University of Minnesota Press.
Camara, W. J., Nathan, J. S., & Puente, A. E. (2000). Psychological
test usage: Implications in professional psychology. Professional
Psychology: Research and Practice, 31(2), 141–154. https://doi.
org/10.1037/0735-7028.31.2.141
Causadias, J. M., Vitriol, J. A., & Atkin, A. L. (2018). Do we overem-
phasize the role of culture in the behavior of racial/ethnic minori-
ties? Evidence of a cultural (mis)attribution bias in American psy-
chology. The American Psychologist, 73(3), 243–255. https://doi.
org/10.1037/amp0000099
Charles, N. E., Tennity, C., Anestis, J. C., Kim, H., & Barry, C. T.
(2022). Evaluating measures of externalizing personality pathology
traits in Black and White American adolescents in a program for
at-risk youths. Journal of Personality Assessment, 1–12. https://doi.
org/10.1080/00223891.2022.2076237
Cohen, J. (1988). Statistical power analyses for the behavioral sciences
(2nd ed.). Lawrence Erlbaum Associates, Inc.
Corey, D. M., & Ben-Porath, Y. S. (2020). Practical guidance on the
use of the MMPI instruments in remote psychological testing.
Professional Psychology: Research and Practice, 51(3), 199–204.
https://doi.org/10.1037/pro0000329
Culpepper, S. A. (2012). Using the criterion-predictor factor model to
compute the probability of detecting prediction bias with ordinary
least squares regression. Psychometrika, 77(3), 561–580. https://doi.
org/10.1007/s11336-012-9270-8
Diedenhofen, B., & Musch, J. (2015). cocor: A comprehensive solution
for the statistical comparison of correlations. PLOS One, 10(3),
e0121945. https://doi.org/10.1371/journal.pone.0121945
Ellis, P. D. (2010). The essential guide to effect sizes: Statistical power,
meta-analysis, and the interpretation of research results. Cambridge
University Press.
Forbey, J. D., Lee, T. T. C., & Handel, R. W. (2010). Correlates of the
MMPI-2-RF in a college setting. Psychological Assessment, 22(4),
737–744. https://doi.org/10.1037/a0020645
16 ANESTIS ET AL.
Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psycho-
logical research: Sense and nonsense. Advances in Methods and
Practices in Psychological Science, 2(2), 156–168. https://doi.
org/10.1177/2515245919847202
Graham, J. R., Veltri, C. O. C., & Lee, T. T. C. (2022). MMPI instru-
ments: Assessing personality and psychopathology (6th ed.). Oxford
University Press.
Gross, D., Fogg, L., Young, M., Ridge, A., Cowell, J. M., Richardson,
R., & Sivan, A. (2006). The equivalence of the Child Behavior
Checklist/1 1/2-5 across parent race/ethnicity, income level, and
language. Psychological Assessment, 18(3), 313–323. https://doi.
org/10.1037/1040-3590.18.3.313
Hall, J. T., Menton, W. H., & Ben-Porath, Y. S. (2022). Examining the
psychometric equivalency of MMPI-3 scale scores derived from the
MMPI-3 and the MMPI-2-RF-EX. Assessment, 29(4), 842–853.
https://doi.org/10.1177/1073191121991921
Han, K., Colarelli, S. M., & Weed, N. C. (2019). Methodological and
statistical advances in the consideration of cultural diversity in
assessment: A critical review of group classification and measure-
ment invariance testing. Psychological Assessment, 31(12), 1481–1496.
https://doi.org/10.1037/pas0000731
Handel, R. W., Archer, R. P., Elkins, D. E., Mason, J. A., &
Simonds-Bisbee, E. C. (2011). Psychometric properties of the
Minnesota Multiphasic Personality Inventory–Adolescent (MMPI–A)
clinical, content, and supplementary scales in a forensic sample.
Journal of Personality Assessment, 93(6), 566–581. https://doi.org/1
0.1080/00223891.2011.608752
Hayes, A. F., Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha
for estimating reliability. But…. Communication Methods and Measures,
14(1), 1–24. https://doi.org/10.1080/19312458.2020.1718629
Ivanova, M. Y., Achenbach, T. M., Rescorla, L. A., Tumer, L. V.,
Ahmeti-Pronaj, A., Au, A., Maese, C. A., Bellina, M., Caldas, J. C.,
Chen, Y.-C., Csemy, L., da Rocha, M. M., Decoster, J., Dobrean,
A., Ezpeleta, L., Fontaine, J. R. J., Funabiki, Y., Guðmundsson, H.
S., Harder, V. S., … Zasepa, E. (2015). Syndromes of self-reported
psychopathology for ages 18–59 in 29 societies. Journal of
Psychopathology and Behavioral Assessment, 37(2), 171–183. https://
doi.org/10.1007/s10862-014-9448-8
Joint Task Force for the Development of Telepsychology Guidelines for
Psychologists. (2013). Guidelines for the practice of telepsychology.
American Psychologist, 68(9), 791–800. https://doi.org/10.1037/
a0035001
Krishnamurthy, R., Hass, G. A., Natoli, A. P., Smith, B. L., Arbisi, P.
A., & Gottfried, E. D. (2022). Professional practice guidelines for
personality assessment. Journal of Personality Assessment, 104(1),
1–16. https://doi.org/10.1080/00223891.2021.1942020
Marek, R. J., Ben-Porath, Y. S., Sellbom, M., McNulty, J. L., & Heinberg,
L. J. (2015). Validity of Minnesota Multiphasic Personality Inventory
- 2 - Restructured Form (MMPI-2-RF) scores as a function of
gender, ethnicity, and age of bariatric surgery candidates. Surgery
for Obesity and Related Diseases : Official Journal of the American
Society for Bariatric Surgery, 11(3), 627–634. https://doi.org/10.1016/j.
soard.2014.10.005
Markus, H. R. (2008). Pride, prejudice, and ambivalence: Toward a
unified theory of race and ethnicity. The American Psychologist,
63(8), 651–670. https://doi.org/10.1037/0003-066X.63.8.651
McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here.
Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/
met0000144
Monnot, M. J., Quirk, S. W., Hoerger, M., & Brewer, L. (2009). Racial
bias in personality assessment: Using the MMPI-2 to predict psy-
chiatric diagnoses of African American and Caucasian chemical
dependency inpatients. Psychological Assessment, 21(2), 137–151.
https://doi.org/10.1037/a0015316
NCS Pearson, I. (2021). Telepractice and questionnaires or rating scales.
Rescorla, L. A., & Achenbach, T. M. (2004). The Achenbach System
of Empirically Based Assessment (ASEBA) for ages 18 to 90+ years.
In M. E. Maruish (Ed.), The use of psychological testing for treatment
planning and outcomes assessment (3rd ed., pp. 115–152). Lawrence
Erlbaum Associates, Inc.
Sellbom, M. (2019). The MMPI-2-Restructured Form (MMPI-2-RF):
Assessment of personality and psychopathology in the twenty-first
century. Annual Review of Clinical Psychology, 15(1), 149–177.
https://doi.org/10.1146/annurev-clinpsy-050718-095701
Sellbom, M. (2021). Examining the criterion and incremental validity
of the MMPI-3 Self-Importance scale. Psychological Assessment,
33(4), 363–368. https://doi.org/10.1037/pas0000975
Sellbom, M., Kremyar, A. J., & Wygant, D. B. (2021). Mapping MMPI-3
scales onto the hierarchical taxonomy of psychopathology.
Psychological Assessment, 33(12), 1153–1168. https://doi.org/10.1037/
pas0001049
Suyemoto, K. L., Erisman, S. M., Holowka, D. W., Fuchs, C.,
Barrett-Model, H., Ng, F., Liu, C., Chandler, D., Hazeltine, K., &
Roemer, L. (2016). UMass Boston comprehensive demographic ques-
tionnaire, revised. The Behavior Therapist, 39, 83–89.
Tellegen, A. (1985). Structures of mood and personality and their
relevance to assessment anxiety, with an emphasis on self-report.
In A. H. Tuma & J. D. Maser (Eds.), Anxiety and the anxiety dis-
orders (pp. 681–706). Erlbaum.
Tellegen, A., Ben-Porath, Y. S., & Sellbom, M. (2009). Construct va-
lidity of the MMPI–2 restructured clinical (RC) scales: Reply to
Rouse, Greene, Butcher, Nichols, and Williams. Journal of Personality
Assessment, 91(3), 211–221. https://doi.org/10.1080/002238
90902794192
Tellegen, A., Ben-Porath, Y., McNulty, J. L., Arbisi, P., Graham, J. R.,
& Kaemmer, B. (2003). MMPI-2 restructured clinical (RC) scales:
Development, validation, and interpretation. Form: technical manu-
al. University of Minnesota Press.
Timbrook, R. E., & Graham, J. R. (1994). Ethnic differences on the
MMPI-2? Psychological Assessment, 6(3), 212–217. https://doi.
org/10.1037/1040-3590.6.3.212
Uliaszek, A. A., Al-Dajani, N., Sellbom, M., & Bagby, R. M. (2019).
Cross-validation of the demoralization construct in the Revised
NEO Personality Inventory. Psychological Assessment, 31(2), 159–166.
https://doi.org/10.1037/pas0000655
University of Minnesota Press (2020). MMPI instruments: Remote ad-
minstration.
Whatley, P. R., Allen, J., & Dana, R. H. (2003). Racial identity and
the MMPI in African American male college students. Cultural
Diversity & Ethnic Minority Psychology, 9(4), 345–353. https://doi.
org/10.1037/1099-9809.9.4.345
Whitman, M. R., & Ben-Porath, Y. S. (2021). Distinctiveness of the
MMPI-3 self-importance and self-doubt scales. Journal of Personality
Assessment, 103(5), 613–620. https://doi.org/10.1080/00223891.2021
.1883628
Whitman, M. R., Tarescavage, A. M., Glassmire, D. M., Burchett, D.,
& Sellbom, M. (2019). Examination of differential validity of
MMPI-2-RF scores by gender and ethnicity in predicting future
suicidal and violent behaviors in a forensic sample. Psychological
Assessment, 31(3), 404–409. https://doi.org/10.1037/pas0000677
Williams, D. R. (2018). Stress and the mental health of populations
of color: Advancing our understanding of race-related stressors.
Journal of Health and Social Behavior, 59(4), 466–485. https://doi.
org/10.1177/0022146518814251
Wright, C. V., Beattie, S. G., Galper, D. I., Church, A. S., Bufka, L. F.,
Brabender, V. M., & Smith, B. L. (2017). Assessment practices of
professional psychologists: Results of a national survey. Professional
Psychology: Research and Practice, 48(2), 73–78. https://doi.
org/10.1037/pro0000086
Yip, T. (2018). Ethnic/racial identity—a double-edged sword?
Associations with discrimination and psychological outcomes.
Current Directions in Psychological Science, 27(3), 170–175. https://
doi.org/10.1177/0963721417739348
Zou, G. Y. (2007). Toward using confidence intervals to compare
correlations. Psychological Methods, 12(4), 399–413. https://doi.org/
10.1037/1082-989X.12.4.399