Applied Neuropsychology: Child

ISSN: 2162-2965 (Print) 2162-2973 (Online) Journal homepage: https://www.tandfonline.com/loi/hapc20

Verbal fluency and digit span variables as

performance validity indicators in experimentally
induced malingering and real world patients with
TBI

Jessica Hurtubise, Tabarak Baher, Isabelle Messa, Laura Cutler, Ayman

Shahein, Maurissa Hastings, Marilou Carignan-Querqui & Laszlo A Erdodi

To cite this article: Jessica Hurtubise, Tabarak Baher, Isabelle Messa, Laura Cutler,
Ayman Shahein, Maurissa Hastings, Marilou Carignan-Querqui & Laszlo A Erdodi (2020):
Verbal fluency and digit span variables as performance validity indicators in experimentally
induced malingering and real world patients with TBI, Applied Neuropsychology: Child, DOI:
10.1080/21622965.2020.1719409

To link to this article: https://doi.org/10.1080/21622965.2020.1719409

Published online: 21 Feb 2020.

Submit your article to this journal

Article views: 10

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=hapc20
APPLIED NEUROPSYCHOLOGY: CHILD
https://doi.org/10.1080/21622965.2020.1719409

Verbal fluency and digit span variables as performance validity indicators in

experimentally induced malingering and real world patients with TBI
Jessica Hurtubisea, Tabarak Bahera, Isabelle Messaa, Laura Cutlera, Ayman Shaheinb, Maurissa Hastingsa, Marilou
Carignan-Querquic, and Laszlo A Erdodia
a
Department of Psychology, University of Windsor, Windsor, Canada; bDepartment of Clinical Neurosciences, University of Calgary, Calgary,
Canada; cDepartment of Languages, Literature and Cultures, University of Windsor, Windsor, Canada

ABSTRACT KEYWORDS
Objective: This study was designed to examine the classification accuracy of verbal fluency (VF) Experimental malingering;
measures as performance validity tests (PVT). performance validity;
Method: Student volunteers were assigned to the control (n ¼ 57) or experimental malingering population-specific cutoffs;
verbal fluency
(n ¼ 24) condition. An archival sample of 77 patients with TBI served as a clinical comparison.
Results: Among students, FAS T-score 29 produced a good combination of sensitivity (.40–.42)
and specificity (.89–.95). Animals T-score 31 had superior sensitivity (.53–.71) at .86-.93 specificity.
VF tests performed similarly to commonly used PVTs embedded within Digit Span: RDS 7
(.54–.80 sensitivity at .93–.97 specificity) and age-corrected scaled score (ACSS) 6 (.54–.67 sensi-
tivity at .94–.96 specificity). In the clinical sample, specificity was lower at liberal cutoffs [animals T-
score 31 (.89–.91), RDS 7 (.86–.89) and ACSS 6 (.86–.96)], but comparable at conservative cut-
offs [animals T-score 29 (.94–.96), RDS 6 (.95–.98) and ACSS 5 (.92–.96)].
Conclusions: Among students, VF measures had higher signal detection performance than previ-
ously reported in clinical samples, likely due to the absence of genuine impairment. The superior
classification accuracy of animal relative to letter fluency was replicated. Results suggest that exist-
ing validity cutoffs can be extended to cognitively high functioning examinees, and emphasize
the importance of population-specific cutoffs.

Performance validity and cognitive testing Types of PVTs

Establishing the credibility of a given response set is critical
to the validity of clinical interpretations based on neuro-
psychological test scores (Fuermaier et al., 2017; Merten &
Rogers, 2017; Stevens, Friedel, Mehren, & Merten, 2008).
Since clinical judgment is notoriously poor at detecting
invalid performance (Dandachi-FitzGerald, Merckelbach, &
Ponds, 2017; Heaton, Smith, Lehman, & Vogt, 1978), per-
formance validity tests (PVTs) have been developed to pro-
vide objective measures of the extent to which a given test
score (or the entire neurocognitive profile) is an accurate
reflection of the examinee’s underlying abilities. Recent sur-
veys have detected a trend toward increased utilization of
validity tests in neuropsychology (Martin, Schroeder, &
Odland, 2015), following longstanding recommendations by
professional organizations (Bush et al., 2005; Bush,
Heilbronner, & Ruff, 2014; Heilbronner et al., 2009).
However, a recent report on the discrepancy between self-
reported and observed PVT use cautions against equating
survey results with actual clinical practice (MacAllister,
Vasserman, & Armstrong, 2019).
There has been a gradual increase in the use of embedded
validity indicators (EVIs) to complement free-standing PVTs
dedicated to discriminating valid and invalid responding.
Later co-opted as PVTs, EVIs are tests developed initially as
measures of cognitive ability. The recent proliferation of
EVI research, both in terms of introducing new instruments
(Berger et al., 2019; Erdodi, Seke, et al., 2017; Erdodi et al.,
2016; Fuermaier et al., 2016; Lichtenstein, Erdodi, & Linnea,
2017; Ord, Boettcher, Greve, & Bianchini, 2010; Rai, An,
Charles, Ali, & Erdodi, 2019; Whiteside, Kogan, et al., 2015)
and the ongoing cross-validation of existing ones (Abeare,
Sabelli, et al., 2019; Erdodi, Pelletier, & Roth, 2018; Lange
et al., 2013; Persinger et al., 2018; Schroeder, Twumasi-
Ankrah, Baade, & Marshall, 2012; Webber & Soble, 2018;
Whiteside, Caraher, Hahn-Ketter, Gaasedelen, & Basso,
2019; Whitney, Davis, Shepard, Bertram, & Adams, 2009) is
a testament to their clinical utility. Add-ons to free-standing
PVTs (post-publication enhancements that are analogous to
EVIs) have also been shown to improve classification accur-
acy (Boone, Salazar, Lu, Warner-Chacon, & Razani, 2002;
Erdodi, Tyson, et al., 2017, 2018; Kim et al., 2010; Lupu,

CONTACT Laszlo Erdodi lerdodi@gmail.com Department of Psychology, University of Windsor, Windsor, Canada.
ß 2020 Taylor & Francis Group, LLC
2 J. HURTUBISE ET AL.
Elbaum, Wagner, & Braw, 2018; Tomer, Lupu, Golan,
Wagner, & Braw, 2019).
Free-standing PVTs vs. EVIs
Unlike most free-standing PVTs, EVIs provide information
on both performance validity and cognitive ability, making
them a cost-effective choice in assessment contexts where
time spent on test administration and scoring is a highly
valued commodity (Bortnik et al., 2010; Erdodi, Kirsch,
Sabelli, & Abeare, 2018; Glassmire, Wood, Ta, Kinney, &
Nitch, 2019). Additionally, EVIs are inseparable from their
host instrument, reducing the inferential leap from free-
standing PVTs to measures of cognitive ability when deter-
mining the credibility of a response set (Bigler, 2015). Since
they are harder to identify, EVIs are also more resistant to
the effect of coaching (Brennan et al., 2009; Kanser et al.,
2017). EVIs also help reduce the appearance of bias toward
malingering detection due to reliance on multiple free-
standing PVTs dedicated solely to the identification of non-
credible response sets (Boone, 2013).
Despite the myriad of advantages, individually, EVIs tend
to have inferior signal detection properties relative to free-
standing PVTs (Erdodi, Green, Sirianni, & Abeare, 2019).
This is particularly problematic in forensic settings, where
false positive errors have far-reaching consequences. More
importantly, EVIs are often criticized for blurring the line
between credible deficits and invalid performance. Although
free-standing PVTs are not immune to this line of reasoning
either (Bigler, 2012; Leighton, Weinborn, & Maybery, 2014),
EVIs are especially vulnerable, at least on theoretical
grounds, to the claim that they conflate genuine impairment
and non-credible responding (Boskovic et al., 2018; Eglit
et al., 2019; Glassmire et al., 2019). While the inherent diffi-
culty of tests can be manipulated effectively through care-
fully planned test construction and/or deliberate choice of
cutoff, all neuropsychological tests require some level of cog-
nitive ability to yield a clinically meaningful score—an
inescapable epistemological challenge for PVT designers and
users (Bigler, 2014; Erdodi, 2019; Lippa, 2018). Ultimately,
no PVT is robust enough to provide meaningful data when
confounded by extreme levels of impairment (Boone, 2013;
Green, Montijo, & Brockhaus, 2011; Larochette & Harrison,
2012). Nevertheless, free-standing PVTs tend to have better
classification accuracy than EVIs (MacAllister et al., 2019).
However, EVIs face an additional, unique challenge: the
invalid before impaired paradox (Erdodi & Lichtenstein,
2017). Namely, several validity cutoffs reach into the range
of scores historically considered to indicate normal cognitive
functioning. At face value, such occurrences undermine the
credibility of EVIs, as they appear to erase the range of cred-
ible impairment. Although a series of arguments have been
marshalled to defend EVIs against such concerns (Hilsabeck,
2017), free-standing PVTs are immune to the invalid before
impaired paradox by default, as they are expected only to
provide information on performance validity. Since they are
typically designed to be easy while appearing difficult and
are passed by the vast majority of credible patients with
genuine and severe deficits (Carone, Green, & Drane, 2014;
Green, Montijo, & Brockhaus, 2011), free-standing PVTs at
sufficiently conservative cutoffs tend to have superior speci-
ficity. Therefore, a score in the failing range on such tests
makes a convincing rational and empirical argument for
invalid performance.
Verbal fluency tests as EVIs
Verbal fluency measures have long been known to be sensi-
tive to non-credible responding. Hayward, Hall, Hunt, and
Zubrick (1987) reported that nurses who were instructed to
feign credible impairment performed significantly more
poorly than patients with medically verified TBI on animal
fluency. The effect size (d ¼ 1.46) was comparable to that
observed on Digit Span (d ¼ 1.51; very large). In contrast,
Backhaus, Fichtenberg, and Hanks (2004) found that,
although invalid performance had a large effect on letter flu-
ency (d ¼ 0.81), it was one of the most robust measures to
non-credible performance. As a reference, a much larger
effect (d ¼ 1.48) emerged on the written version of the
Symbol-Digit Modalities Test.
These findings were later replicated by the same research
group: non-credible responding had a large effect (d ¼ 0.83)
on letter fluency scores (Johnson, Silverberg, Millis, &
Hanks, 2012). More recently, Whiteside, Gaasedelen, et al.
(2015) reported a medium-large effect for psychometrically
defined invalid performance on animal fluency (d ¼ 0.65)
among patients with mild TBI. In addition, Johnson et al.
(2012) combined the total score on a different version of the
letter fluency test (CFL) with the change in output over
time in a logistic regression equation. The standard cutoff
(.50) produced a good combination of sensitivity (.67) and
specificity (.88). Increasing the cutoff to .60 disproportion-
ately sacrificed sensitivity (.43) for specificity (.89).
In light of accumulating evidence that verbal fluency
measures had the potential to serve as EVIs, formal validity
cutoffs were first introduced by Curtis, Thompson, Greve,
and Bianchini (2008). They proposed that a demographically
adjusted T-score of 33 (in mild TBI) and 30 (moderate-
to-severe TBI) achieved .90 specificity at .14–.34 sensitivity.
Their findings were replicated by Whiteside, Kogan, et al.
(2015), who concluded that a more conservative cutoff
(T  24) on letter fluency was needed to maintain .90 speci-
ficity. However, this cutoff was insensitive to invalid per-
formance (.05). Animal fluency cutoffs (T  25 and T  24)
produced slightly better combinations of sensitivity (.23–.25)
and specificity (.89–.91).
The superior signal detection performance of animal ver-
sus letter fluency was also observed in a study by Sugarman
and Axelrod (2015). In their large sample of clinically
referred veterans, a T-score of 30 on FAS produced .30
sensitivity at .90 specificity. More liberal cutoffs (T  33 and
T  31) achieved a better combination of sensitivity
(.42–.44) and specificity (.89–.91) on animal fluency. Table 1
provides a quick visual summary of the literature
reviewed above.
 APPLIED NEUROPSYCHOLOGY: CHILD 3

Table 1. Brief summary of the classification accuracy of verbal fluency measures as PVTs.
Study Sample characteristics Criterion Test Cutoff SENS SPEC
Curtis et al., 2008 204 patients with traumatic brain Slick, Sherman, and Mild TBI
injury (TBI); MAge ¼ 39.6; Iverson (1999) FAS T  33 .34 .92
MEducation ¼ 12.3 criteria for MND T  31 .21 .97
Moderate-to-Severe TBI
FAS T  33 .29 .79
T  31 .19 .87
T  30 .14 .91
T  28 .14 .95
Whiteside, Kogan, 57 compensation-seeking patients Valid: 0 PVT failures; FAS T  25 .09 .87
et al., 2015 with mild TBI & 61 non- Invalid: 2 T  24 .05 .90
compensation-seeking patients PVT failures T  17 .02 1.00
with moderate-to-severe TBI Animals T  25 .25 .89
T  24 .23 .91
T  23 .19 .94
Sugarman & 623 physician-referred VA patients; Valid: 0 PVT failures; FAS T  30 .30 .90
Axelrod, 2015 MAge–Valid ¼ 49.6; MEducation–Valid ¼ Invalid: 2 T  29 .26 .91
13.1; MAge–Invalid ¼ 51.3; PVT failures Animals T  33 .44 .89
MEducation– Invalid ¼ 12.6 T  31 .42 .91
Note. PVT: Performance validity test; MND: Malingered Neurocognitive Dysfunction (Slick et al., 1999); SENS: Sensitivity; SPEC: Specificity

The review of existing evidence on verbal fluency measures
as EVIs converges on several conclusions. First, there is sig-
nificant variability in cutoffs both across and within studies.
When compared directly, animal fluency cutoffs outper-
formed letter fluency cutoffs. Second, once .90 specificity
was achieved, sensitivity tended to be low and variable. Third,
TBI severity was related to the signal detection profile of the
verbal fluency based EVIs: more conservative cutoffs were
required in patients with moderate-to-severe TBI to maintain
the same level of specificity compared to patients with mild
TBI (Curtis et al., 2008). Finally, performance on letter and
category fluency tests varied across studies as a function of
sample characteristics, stimulus presentation (Silverberg,
Hanks, Buchanan, Fichtenberg, & Millis, 2008), study design,
and criterion grouping (Crowe, 1996). These findings under-
score the importance of developing population-specific cut-
offs (Glassmire et al., 2019; Pearson, 2009) as a potential
practical solution to divergent evidence on the classification
accuracy of EVIs within measures of verbal fluency.
Current study
Combined with recent reports suggesting that PVTs tend to
be insensitive to non-credible performance in cognitively
intact individuals (Abeare, Messa, et al., 2019; An, Kaploun,
Erdodi, & Abeare, 2017; Roye, Calamia, Bernstein, De Vito,
& Hill, 2019), the current state of knowledge calls for a rep-
lication of validity cutoffs on verbal fluency measures in a
non-clinical sample. This study was designed to use an
experimental malingering (expMAL) paradigm to examine
the classification accuracy of existing cutoffs within a non-
clinical student sample. Given the paucity of research on
EVIs nested within verbal fluency tests (Curtis et al., 2008;
Johnson et al., 2012; Whiteside, Kogan, et al., 2015), the
analyses were extended to commonly used Digit Span based
EVIs, a well-researched instrument in the context of per-
formance validity assessment.
Given that free-standing PVTs may be insensitive to sub-
tle manifestations of non-credible responding in high
functioning examinees while simultaneously conflating
invalid performance with genuine deficits in clinical patients,
relying on healthy, young, and cognitively high-functioning
participants serves as a potentially valuable proof of concept.
Since such a design controls for genuine deficits, scores
below the validity cutoff have a clearer interpretation (i.e.,
non-credible responding). On the other hand, EVI cutoffs
calibrated on a cognitively high functioning sample (Abeare,
Freund, Kaploun, McAuley, & Dumitrescu, 2017; Erdodi,
Sagar, et al., 2018) often fail to replicate in patients with
more severe deficits (Eglit et al., 2019; Glassmire et al.,
2019). Therefore, the classification accuracy statistics in stu-
dents were compared to a clinical sample of patients
with TBI.
We hypothesized that previously published cutoffs (FAS
and animals T-score 33) would produce better classifica-
tion accuracy in university students who volunteered as
research participants compared to the clinical samples, as
the signal detection model would not be contaminated by
genuine cognitive impairment as a confounding variable
(Berger et al., 2019; Bodner, Merten, & Benke, 2019; Rai
et al., 2019). Similarly, we predicted that verbal fluency
based EVIs would be sensitive to injury severity. Finally, we
hypothesized an inverse dose-response relationship between
failure rates on other PVTs and TBI severity (mild vs mod-
erate/severe), consistent with previous reports (Abeare,
Sabelli, et al., 2019; Erdodi & Rai, 2017; Green et al., 2011).
Method
Participants
Student sample
Participants were recruited through the University’s online
research platform and were randomly assigned to either the
control or the expMAL condition, following a 2:1 ratio.
Inclusion criteria for this study were age 18–35, absence of
major neurological disorders, and enrollment in an under-
graduate psychology or business course. There were no dif-
ferences between the two groups in age [MControl ¼ 20.6,
4 J. HURTUBISE ET AL.

Table 2. List of neuropsychological tests administered to the student sample (n ¼ 81).

Name of the test Abbreviation Reference Norms
Animal Fluency Animals Gladsjo et al., 1999 Heaton
Boston Naming Test—Short Form (Abbreviated) BNT-15 Goodglass, Kaplan, & Barresi, 2001; Erdodi, Jongsma, et al., 2017 Manual
Complex Ideational Material CIM Goodglass et al., 2001 Heaton
Digit Span DSWAIS-III Wechsler, 1997 Manual
Digit-Symbol Coding CDWAIS-III Wechsler, 1997 Manual
Grooved Pegboard Test GPB Matthews & Klove, 1964 Heaton
Hopkins Verbal Learning Test—Revised HVLT-R Brandt & Benedict, 2001 Manual
Letter Fluency FAS Gladsjo et al., 1999 Heaton
Rey 15-Item Test with Recognition Rey-15 Rey, 1941; Boone, Salazar, Lu, Warner-Chacon, & Razani, 2002 –
Wisconsin Card Sorting Test—64 Card Version WCST-64 Kongs, Thompson, Iverson, & Heaton, 2000 Manual
Single Word Reading WRAT-4 Wilkinson & Robertson, 2006 Manual
Symbol-Digit Modalities Test SDMT Smith, 2007 Manual
Trails 2 and 4 D-KEFS Delis, Kaplan, & Kramer, 2001 Manual
Word Choice Test WCT Pearson, 2009 –
Note. WAIS-III: Wechsler Adult Intelligence Scale—Third Edition; WRAT-4: Wide Range Achievement Test—Fourth Edition; D-KEFS: Delis–Kaplan Executive Function
System; Heaton: Demographically adjusted norms published by Heaton, Miller, Taylor, and Grant (2004).

Table 3. Components of the VI-7, cutoffs and corresponding base rates of failure in the student sample (n ¼ 81).
VI-7 Component Scale Cutoff BRFail Reference
BNT-15 T2C Raw 85” 9.9 An et al., 2019; Erdodi, Dunn, et al., 2018
CDWAIS-III ACSS 5 14.8 Ashendorf, Clark, & Sugarman, 2017; Erdodi, Abeare et al., 2017; Erdodi & Lichtenstein, 2017; Trueblood, 1994;
CIMBDAE Raw 9 27.2 An et al., 2019; Erdodi, 2019; Erdodi & Roth, 2017; Erdodi et al., 2016
FMSWCST-64 Raw 2 8.6 Greve, Bianchini, Mathias, Houston, & Crouch, 2002; Lichtenstein, Holcomb, et al., 2018; Suhr & Boyer, 1999
GPB DH T 29 27.2 Erdodi, Kirsch, et al., 2018; Erdodi, Seke, et al., 2017
RDHVLT-R Raw 6 16.0 Bailey, Soble, Bain, & Fullen, 2018; Sawyer, Testa, & Dux, 2017
Trails 2D-KEFS ACSS 5 17.3 Erdodi, Hurtubise, et al., 2018; Erdodi & Lichtenstein, 2019
Note. VI-7: Validity Index Seven; BNT-15: Boston Naming Test—Short Form; T2C: Time to completion; CDWAIS-III : Coding subtest of the Wechsler Adult Intelligence
Scale—Third Edition; CIMBDAE: Complex Ideational Material subtest of the Boston Diagnostic Aphasia Battery; FMSWCST : Failures to Maintain Set on the 64-card
version of the Wisconsin Card Sorting Test; GPB DH: Grooved Pegboard Test—Dominant hand; RDHVLT-R: Recognition Discrimination Index (Yes/No recognition
trial true positives (hits) minus false positives); Trails 2D-KEFS: Trails 2 subtest (Number Sequencing) on the Delis–Kaplan Executive Function System.

SD ¼ 2.6; MexpMAL ¼ 21.0, SD ¼ 3.0; t(79) ¼ 0.66, p ¼ .514], Materials

education [MControl ¼ 14.3, SD ¼ 1.2; MexpMAL ¼ 14.7,
SD ¼ 1.2; t(79) ¼ 1.53, p ¼ .129] or proportion of females
[84% vs 83%, v2(1) ¼ 0.08, p ¼ .785].
Clinical sample
To provide a direct comparison to patients with genuine
neuropsychological deficits, the classification accuracy of the
EVIs of interest was recomputed in a sample of 77 patients
(57.1% male) clinically referred for neuropsychological test-
ing following a TBI at a nearby academic medical center.
The majority of the injuries (75.3%) were classified as mild
based on available injury parameters (Glasgow Coma Scale,
duration of loss of consciousness and peri-traumatic
amnesia, intracranial abnormalities on neuroradiological
findings). The remaining 24.7% were classified as moderate/
severe. Patients were selected from a consecutive case
sequence used in previous publications by the same research
group (Abeare, Sabelli et al., 2019; Erdodi & Abeare, 2019;
Erdodi, Abeare, Medoff et al., 2018; Erdodi, Roth et al.,
2014; Erdodi, Taylor et al., 2019). Inclusion criteria were age
50 or younger, data available on animal fluency and the
same free-standing PVT used in the student sample. Mean
age was 32.7 years (SD ¼ 10.5), while the mean level of edu-
cation was 13.3 years (SD ¼ 2.4). All patients were in the
post-acute stage of recovery (>3 months since a mild TBI
and >12 months since a moderate-severe TBI) and evaluated
in an outpatient setting.
Student sample
Table 2 provides a list of neuropsychological tests adminis-
tered to the student sample (n ¼ 81). The Rey-15 and Word
Choice Test (WCT) were the two free-standing PVTs. In
addition, seven EVIs were aggregated into a single compos-
ite labeled “Validity Index Seven” (VI-7). Components of
the VI-7 are presented in Table 3, along with references to
the cutoffs used. The value of the VI-7 is the cumulative
number of failures across all seven components. As such,
possible scores range from 0 (all seven PVTs passed) to 7
(all seven PVTs failed). The majority of the sample (72.8%)
scored in the Pass range (1) on the VI-7; a small propor-
tion (6.2%) scored in the Borderline range (2), and one fifth
(21.0%) scored in the Fail range (3). To maintain the pur-
ity of the criterion groups, participants in the Borderline
range were excluded from analysis in which the VI-7 served
as the dichotomous (Pass/Fail) reference PVT, consistent
with emerging practice standards (Abeare, Messa, et al.,
2019; Axelrod, Meyers, & Davis, 2014; Erdodi, 2019;
Schroeder, Martin, Heinrichs, & Baade, 2019; Sugarman &
Axelrod, 2015; Whiteside, Kogan, et al., 2015).
Clinical sample
Patients were administered a comprehensive battery of
neuropsychological tests. However, data on FAS and certain
Digit Span variables (longest span forward and backward)
were not available. As both samples were administered the
 APPLIED NEUROPSYCHOLOGY: CHILD 5

Table 4. Components of the VI-7, cutoffs and corresponding base rates of failure in the clinical sample (n ¼ 77).
VI-7 Component Scale Cutoff BRFail Reference
CDWAIS-IV ACSS 5 26.0 Ashendorf et al., 2017; Erdodi & Abeare, 2019; Erdodi, Abeare et al., 2017; Erdodi & Lichtenstein, 2017; Trueblood, 1994
FCRCVLT-II Raw 15 24.7 Erdodi, Abeare et al., 2018; Persinger et al., 2018; Schwartz et al., 2016
FMSWCST Raw 2 11.8 Greve et al., 2002; Lichtenstein, Holcomb, et al., 2018; Suhr & Boyer, 1999
FTT DH T 31 11.7 Erdodi, Taylor, et al., 2019
GPB DH T 29 25.0 Erdodi, Kirsch, et al., 2018; Erdodi, Seke, et al., 2017
LNSWAIS-IV ACSS 7 25.7 Erdodi & Abeare, 2019; Shura et al., 2016
SSWAIS-IV ACSS 6 29.9 Erdodi, Abeare et al., 2017; Erdodi & Abeare, 2019; Trueblood, 1994
Note. VI-7: Validity Index Seven; CDWAIS-IV: Coding subtest of the Wechsler Adult Intelligence Scale—Fourth Edition; FMSWCST: Failures to Maintain Set on the
Wisconsin Card Sorting Test; GPB DH: Grooved Pegboard Test—Dominant hand; ACSS: Age-corrected scaled score.

WCT, it was used as the shared criterion PVT. Additionally,
a composite validity index was created mirroring the VI-7,
based on EVIs available in the data set (Table 4). The
majority of the sample (58.4%) scored in the Pass range
(1) on the VI-7. A small proportion (13.0%) scored in the
Borderline range (2), and 28.6% scored in the Fail
range (3).
Procedure
Student sample
Testing was completed in a quiet, distraction-free room.
Informed consent was obtained from all participants prior
to psychometric testing. Participants assigned to the
expMAL condition were provided a script that instructed
them to feign credible cognitive impairments in the fictional
context of personal injury litigation in which they were the
victim of a motor vehicle collision. The scenario has been
previously used by this research group (An et al., 2019; Rai
et al., 2019) and was modeled after comparable vignettes
developed by DenBoer and Hall (2007).
At the end of testing, all participants completed a
manipulation check. They were presented with a paper-and-
pencil questionnaire that included the open-ended prompt
“explain what you were asked to do in this study” and a
request to rate their compliance to task instructions on a
scale from 0 to 10. Participants in the expMAL condition
had additional items. First, they were asked to rate how well
they could relate to the imagined initial script on a scale
from 0 to 10. Second, they were asked to select which strat-
egies they used to pretend to be impaired during testing
from a variety of options (including “I didn’t pretend”). The
study was approved by the University’s Research
Ethics Board.
Data analysis
Descriptive statistics [M, SD, base rates of PVT failure
(BRFail), risk ratios (RR)] were reported where relevant. The
inferential statistics were independent samples t-tests and v2.
The assumption of homogeneity of variance was examined
with Levene’s test. Effect size estimates were expressed in
Cohen’s d and U2. Given that studies based on the expMAL
paradigm tend to produce predictably inflated effect sizes,
the lower limit for moderate effect has been redefined as
.75; for large effect as 1.25, and for very large as 1.75
(Rogers, Sewell, Martin, & Vitacco, 2003). Receiver operat-
ing characteristics [area under the curve (AUC), 95% CIs]
was computed in SPSS version 25.0. AUC values in the
.70–.79 range are considered acceptable, whereas values .90
are considered outstanding (Hosmer & Lemeshow, 2000).
Sensitivity and specificity values were calculated using stand-
ard formulas. The minimum acceptable level of specificity is
.84 (Larrabee, 2003), but values .90 are desirable and
becoming the emerging norm (Boone, 2013).
To examine the potential contribution of process varia-
bles within verbal fluency to differentiating valid from
invalid responding, logistic regression classifiers (LRC) were
generated using MATLAB’s generalized-linear-model pack-
age glmfit. They were cross-validated via n ¼ 1000 repeat
cross-validation on an 80:20 training-set versus testing-set
split with binomial classification using expMAL status as the
criterion. Step-wise analysis was performed based on an
embedded t-test analysis testing the null-hypothesis that the
parameter coefficients do not significantly differ from zero
(i.e., offer no significant contribution to LRC performance).
Results
Comparing the two samples: demographic
characteristics and neuropsychological profiles

Clinical sample As anticipated, the student sample was significantly younger

(d ¼ 1.57, very large effect) and more educated (d ¼ 0.57,
Patients were evaluated at the outpatient neurorehabilitation medium effect). Also, a significantly lower proportion of stu-
service of a large academic medical center. dents was male (14.8%) compared to the clinical sample
Neuropsychological tests were administered and scored by (57.1%). Although single word reading performance was in
Master’s level psychometrists under the supervision of a the Average range in both samples, the students had a sig-
licensed clinical neuropsychologist. The evaluation was com- nificantly higher mean score (d ¼ 0.56, medium effect).
pleted in two half-day (i.e., four-hour) appointments. Only Likewise, the student sample scored above the TBI sample
de-identified data were collected for research purposes. The on most neuropsychological tests (Table 5). Interestingly,
project was approved by the Institutional Review Board of there was no difference in animal fluency performance
the hospital. between the two samples.
6 J. HURTUBISE ET AL.

Table 5. Comparing the student and clinical samples on demographic and neuropsychological variables.
Sample
Student Clinical
Variable n M SD n M SD t p d
Age 81 20.7 2.7 77 32.7 10.5 9.95 <.001 1.57
Education 81 14.4 1.3 77 13.3 2.4 3.61 <.001 0.57
% Male 81 14.8 77 57.1 5.56 <.001 –
ReadingWRAT-4 57 100.8 13.6 77 93.6 12.2 3.22 .002 0.56
WCT 57 49.6 1.0 77 47.8 3.6 3.67 <.001 0.72
RDS 57 9.9 1.6 77 9.2 2.2 2.04 .044 0.36
Animals T-score 57 44.7 9.5 77 42.3 12.5 1.21 .227 –
GPB DH 57 43.3 11.9 77 38.8 13.0 2.05 .042 0.36
DSWAIS-III & IV 57 10.0 2.5 77 8.7 2.8 2.78 .006 0.49
CDWAIS-III & IV 57 10.9 2.8 77 8.0 3.4 5.25 <.001 0.93
Note.  Scores on neuropsychological tests within the student sample were only reported for the control group; ReadingWRAT-4: Single word reading subtest of
the Wide Range Achievement Test—Fourth Edition; WCT: Word Choice Test; RDS: Reliable digit span; GPB DH: Grooved Pegboard Test—Dominant hand (demo-
graphically adjusted T-score); DSWAIS-III & IV: Age-corrected scaled score of the Digit Span subtest of the Wechsler Adult Intelligence Scale—Third (Students) and
Fourth (Patients) Editions; CD: Coding age-corrected scaled score.

Table 6. The effect of experimental malingering on Verbal Fluency, Digit Span, PVTs and Select Neuropsychological Tests in the student sample (n ¼ 81).
Criterion group
Control expMAL
n ¼ 57 n ¼ 24
Test/Variable M SD M SD t p d r12 vs. r22
FAS 42.1 9.3 31.8 10.0 4.47 <.001 1.07 .746
Animals 44.7 9.5 29.6 14.5 5.50 <.001 1.23 .011
LDF 5.9 1.0 3.9 2.0 6.01 <.001 1.26 <.001
LDB 4.1 0.9 2.7 1.4 5.03 <.001 1.19 .074
RDS 9.9 1.6 6.6 3.2 6.15 <.001 1.30 <.001
ACSSDigit Span 10.0 2.5 6.3 3.3 5.47 <.001 1.26 .032
LDF–LDB 1.8 1.1 1.2 1.2 2.45 .017 0.62 .309
WCT 49.6 1.0 40.2 7.8 8.98 <.001 1.69 <.001
Rey-15 28.9 2.0 21.1 8.6 6.44 <.001 1.25 <.001
VI-7 0.5 0.9 3.0 2.2 7.33 <.001 1.49 <.001
ReadingWRAT-4 100.8 13.6 94.7 17.8 1.67 .099 0.39 .169
CATWCST-64 3.7 1.1 2.4 1.7 4.09 .001 0.91 .004
SDMTWritten 1.62 1.05 1.32 2.01 8.26 <.001 1.83 <.001
HVLT-R 1-3 25.9 3.8 21.0 7.6 3.88 <.001 0.82 <.001
HVLT-R DR 8.8 2.1 5.3 3.2 5.77 <.001 1.29 .004
Trails 4D-KEFS 9.4 2.8 6.8 3.8 3.43 .001 0.78 .008
Note. LDF: Longest digits forward; LDB: Longest digits backward; RDS: Reliable Digit Span; ACSS: Age-corrected scaled score (M ¼ 10, SD ¼ 3); ReadingWRAT-4:
Single Word Reading subtest on the Wide Range Achievement Test—Fourth Edition standard score (M ¼ 100, SD ¼ 15); CATWCST-64: Categories completed on
the Wisconsin Card Sorting Test—64 Card Version; SDMT: Symbol-Digit Modalities Test z-score (M ¼ 0.00, SD ¼ 1.0); HVLT-R: Hopkins Verbal Learning Test-
Revised (raw scores); Trails 4D-KEFS: Trails 4 (Number-Letter Switching) on the Delis-Kaplan Executive Function System ACSS; expMAL: Experimental malingering;
r12 vs. r22: The p-value associated with Levene’s test of homogeneity of variance.

The effect of expMAL on verbal fluency and digit span
performance in the student sample
Participants in the control group scored significantly higher
on all measures compared to those in the expMAL condition
(Table 6). Effect sizes ranged from medium [d ¼ .62; longest
digit span forward (LDF) minus longest digit span backward
(LDB)] to large [d ¼ 1.30; reliable digit span (RDS)]. In add-
ition, expMAL was associated with significantly higher
within-group variability on animal fluency, LDF, RDS, and
Digit Span age-corrected scaled scores (ACSS).
Predictably, performance among participants in the
expMAL condition was significantly below the normative
mean on both letter (d ¼ 1.82, very large) and animal flu-
ency (d ¼ 1.64, very large). Likewise, participants scored
below the norms on Digit Span ACSS (d ¼ 1.10, moderate
effect). Interestingly, even the control group scored signifi-
cantly lower than the normative mean on both letter
(d ¼ 0.81, large) and animal fluency (d ¼ 0.54, medium
effect). However, they performed within the expected range
on Digit Span ACSS (Table 6).
The effect of expMAL on free-standing/composite PVTs
and tests of cognitive ability in the student sample
A large effect was observed on all three criterion PVTs
(d: 1.25–1.69). Participants in the expMAL also produced
significantly higher within-group variability. The difference
on the single word reading test was non-significant.
However, significant contrasts emerged on tests of concept
formation (d ¼ 0.91, moderate effect), graphomotor process-
ing speed (d ¼ 1.83, very large effect), visuomotor scanning,
and switching (d ¼ 0.78, moderate effect). On an auditory
verbal learning test, there was a moderate effect (d ¼ 0.82)
on the acquisition trials and a large effect on the delayed
free recall trial (d ¼ 1.29). As before, expMAL was associated
with significantly higher within-group variability.
 APPLIED NEUROPSYCHOLOGY: CHILD 7

Table 7. Overall classification accuracy of Verbal Fluency and Digit Span Variables against criterion measures in the student sample (n ¼ 81).
Criterion measure
expMAL WCT Rey-15 VI-7
Test/Variable AUC 95% CI AUC 95% CI AUC 95% CI AUC 95% CI
FAS .77 .65–.89 .68 .53–.83 .64 .47–.81 .64 .48–.81
Animals .80 .69–.91 .88 .78–.98 .74 .58–.90 .86 .74–.98
LDF .81 .71–.92 .90 .81–.99 .83 .68–.98 .83 .69–.96
LDB .79 .68–.91 .84 .73–.96 .86 .76–.97 .75 .60–.90
RDS .82 .71–.94 .90 .80–.99 .87 .74–.99 .80 .66–.95
ACSS .80 .89–.91 .89 .81–.98 .82 .68–.95 .80 .65–.95
LDF–LDB .64 .51–.77 .71 .58–.85 .64 .46–.81 .71 .57–.85
Note. LDF: Longest digits forward; LDB: Longest digits backward; RDS: Reliable Digit Span; ACSS: Age-corrected scaled score; expMAL: Experimental malingering;
WCT: Word Choice Test (Pearson, 2009; Fail defined as 45; Barhon, Batchelor, Meares, Chekaluk, & Shores, 2015; Bain & Soble, 2019; Davis, 2014; Erdodi et al.,
2014; Erdodi & Lichtenstein, 2019; Zuccato, Tyson, & Erdodi, 2018); Rey-15: Rey Fifteen Item Test (Rey, 1941) combination score (free recall þ recognition hits;
Fail defined as 23; Boone et al., 2002; Poynter et al., 2019); VI-7: Validity Index Seven (Fail defined as 3; Erdodi, Kirsch, et al., 2018; Lichtenstein, Flaro,
et al., 2019).

Table 8. Classification accuracy of Select Verbal Fluency and Digit Span cutoff scores in the student sample (n ¼ 81).
Criterion measure
expMAL WCT Rey-15 VI-7
29.6 21.2 18.8 21.0
Test/Variable Cutoff BRFail SENS SPEC SENS SPEC SENS SPEC SENS SPEC
FAS 33 28.7 .58 .84 .47 .76 .47 .75 .47 .76
31 22.5 .54 .91 .41 .82 .40 .81 .41 .83
29 16.3 .42 .95 .41 .90 .40 .89 .41 .91
27 8.8 .29 1.00 .29 .97 .27 .95 .24 .95
Animals 33 23.8 .58 .91 .76 .90 .60 .84 .81 .91
31 21.3 .54 .93 .71 .92 .53 .86 .71 .93
29 15.0 .42 .96 .59 .97 .53 .94 .59 .98
LDF 4 18.8 .50 .95 .71 .95 .73 .95 .65 .95
3 11.3 .38 1.00 .47 .98 .53 .98 .47 1.00
LDB 2 3.8 .38 1.00 .47 .98 .53 .98 .47 .98
RDS 7 18.8 .54 .96 .71 .95 .80 .97 .59 .93
6 11.3 .38 1.00 .47 .98 .53 .98 .47 1.00
5 10.0 .33 1.00 .47 1.00 .47 .98 .47 1.00
ACSS 6 18.8 .54 .96 .65 .94 .67 .94 .65 .95
5 11.3 .38 1.00 .47 .98 .53 .98 .47 1.00
4 10.0 .33 1.00 .41 .98 .47 .98 .41 1.00
LDF–LDB 0 18.8 .33 .88 .35 .85 .33 .84 .41 .88
–1 1.3 .04 1.00 .06 1.00 .07 1.00 .06 1.00
Note. BRFail: Base rate of failure (% of the sample that failed a given cutoff); expMAL: Experimental malingering; WCT: Word Choice Test (Pearson, 2009; Fail
defined as 45; Barhon et al., 2015; Bain & Soble, 2019; Davis, 2014; Erdodi et al., 2014; Erdodi & Lichtenstein, 2019; Zuccato, Tyson, & Erdodi, 2018); Rey-15:
Rey Fifteen Item Test (Rey, 1941) combination score (free recall þ recognition hits; Fail defined as 23; Boone et al., 2002; Poynter et al., 2019); VI-7: Validity
Index Seven (Fail defined as 3; Erdodi, Kirsch, et al., 2018; Lichtenstein, Flaro, et al., 2019); SENS: Sensitivity; SPEC: Specificity.

ROC curves for verbal fluency and digit span variables combination of sensitivity (.4–.42) and specificity (.89–.95).
in the student sample Making the cutoff more conservative (27) disproportion-
ately sacrificed sensitivity (.24–.29) for specificity (.95–1.00).
The FAS was a significant predictor of expMAL and Pass/
An animal fluency T-score of 33 achieved minimum
Fail status on the WCT, but not Rey-15 and the VI-7
specificity (.84–.91) against all criterion PVTs, with respect-
(Table 7). In contrast, animal fluency produced significant
able sensitivity (.58–.81). Lowering the cutoff to 31
AUCs (.74–.88) against all criterion measures. LDF, LDB,
resulted in the predictable tradeoff: improved specificity
RDS and Digit Span ACSS were significant predictors of all
(.86–.93) and decreased sensitivity (.53–.71). Making the cut-
criterion measures (AUCs: .75–.90). However, LDF–LDB
off more conservative (29) resulted in a balanced recalibra-
had significantly lower AUC values (.64–.71) and failed to
tion of sensitivity (.42–.59) and specificity (.94–.98).
reach significance against the Rey-15.

Relative contribution of process variables and

Classification accuracy of verbal fluency cutoffs in the
student sample
The FAS T-score 33 and 31 cutoffs failed to reach min-
imum specificity against the WCT, Rey-15, and VI-7
(Table 6). Lowering the cutoff to 29 achieved a good
multivariate models in the student sample
In an attempt to replicate the findings of Johnson et al.
(2012), we first examined whether the decline in output
offered any additional benefit over total scores in predicting
expMAL status. Then we explored whether combining both
8 J. HURTUBISE ET AL.

Table 9. Classification accuracy of Select Verbal Fluency and Digit Span cutoff scores in the clinical sample (n ¼ 77).
Criterion measure
WCT VI-7
45 3
19.5 32.8
Test/Variable Cutoff BRFail SENS SPEC SENS SPEC
Animals T-score AUC .74 .74
95% CI .63–.86 .61–.87
33 23.4 .33 .79 .36 .82
31 14.3 .27 .89 .27 .91
29 11.7 .27 .94 .27 .96
27 10.4 .27 .94 .23 .96
25 6.4 .20 .97 .14 .96
AUC .77 .78
95% CI .64–.91 .66–.89
Reliable Digit Span 7 20.8 .47 .86 .27 .89
6 9.1 .27 .95 .18 .98
5 3.9 .07 .95 .05 .98
AUC .83 .85
95% CI .72–.94 .75–.95
Digit Span ACSS 6 22.1 .53 .86 .50 .96
5 14.3 .40 .92 .27 .96
4 9.1 .27 .95 .14 .96
Note. BRFail : Base rate of failure (% of the sample that failed a given cutoff); WCT: Word Choice Test (Pearson, 2009; Fail defined as 45; Barhon et al., 2015;
Bain & Soble, 2019; Davis, 2014; Erdodi et al., 2014; Erdodi & Lichtenstein, 2019; Zuccato, Tyson, & Erdodi, 2018); VI-7: Validity Index Seven (Fail defined as 3;
Erdodi, Kirsch, et al., 2018; Lichtenstein, Flaro, et al., 2019); SENS: Sensitivity; SPEC: Specificity; ACSS: Age-corrected scaled score; AUC: Area under the curve.

letter and category fluency metrics improved signal detection
accuracy over each task being used in isolation, employing a
stepwise LRC and subsequent ROC analysis. The number of
correct responses within the 60-s time limit was split into
four 15-s intervals. The difference between the first
[0–15 sec] and second quarter [16–30 sec] was used as an
estimate for the decline in response output over time. The
discrepancy between these two intervals was chosen because
they offered the largest and most consistent decline in out-
put, maximizing signal-to-noise ratio for classification while
preserving the metric’s simplicity for potential use in the
clinical setting.
The following parameters were loaded into the first LRC:
animals total score, FAS total score, animals declining output,
and FAS declining output. Subsequent t-tests on the LRC
coefficients were only significant for animals (p < 0.012) and
FAS (p < 0.045) total score. Therefore, the declining output
parameters were discarded in a stepwise fashion. A new
LRC was generated with the animals total score and the FAS
total score combined to explore the potential benefit of
aggregating the two metrics. Overall classification accuracy
for the combined LRC was 77.6%—a negligible advantage
over the model based on animals or FAS
alone (75.1–74.6%).
Classification accuracy of digit span cutoffs in the
student sample
An LDF cutoff 4 produced uniformly high specificity (.95)
and moderate sensitivity (.50–.73). Making the cutoff more
conservative (3) disproportionately traded sensitivity
(.38–.53) for specificity (.98–1.00). The only LDB cutoff that
achieved a reasonable balance of sensitivity (.38–.53) and
specificity (.98–1.00) was 2 (Table 8).
The first level of failure on the RDS (7) produced a
good combination of sensitivity (.54–.80) and specificity
(.93–.97). Making the cutoff more conservative (6) sacri-
ficed much of the sensitivity (.38–.53) for small gains in spe-
cificity (.98–1.00). Further lowering the cutoff (5) reached
the point of diminishing return (.33–.47 sensitivity at
.98–1.00 specificity).
An ACSS of 6 achieved a good balance between sensi-
tivity (.54–.67) and specificity (.94–.96). Making the cutoff
more conservative (5) resulted in the predictable improve-
ment in specificity (.98–1.00) at a reasonable cost to sensitiv-
ity (.38–.53). However, further lowering the cutoffs
sacrificed some of the sensitivity (.33–.47) for no gain in
specificity (.98–1.00).
As it was predictable from its poor ROC characteristics,
cutoffs on the derivative EVI (LDF–LBD) struggled to differ-
entiate valid from invalid response sets. The first cutoff to
reach minimum specificity (.84–.88) was 0, with modest
sensitivity (.33–.41). Making the cutoff more conservative
( 1) achieved perfect specificity, but negligible sensitiv-
ity (.04–.07).
Classification accuracy of animal fluency and digit span
cutoffs in the clinical sample
Animal fluency T-scores, RDS and Digit Span ACSS pro-
duced significant AUC against both the WCT and VI-7
(.74–.85). Animals T  33 failed to achieve minimum specifi-
city (.79–.82). Lowering the cutoff to T  31 improved speci-
ficity (.89–.91), at .27 sensitivity. Making the cutoff even
more conservative (T  29) achieved excellent specificity
(.94–.96) without compromising sensitivity (Table 9). The
liberal RDS cutoff (7) cleared the minimum specificity
threshold (.86–.89) at .27–.47 sensitivity. The conservative
 APPLIED NEUROPSYCHOLOGY: CHILD 9

Table 10. Base rates of PVT failure as a function of TBI severity in the clinical specificity fixed at .90 level. Likewise, on animal fluency,
sample (n ¼ 77).
higher sensitivity (.53–.81 versus .23–.42) was observed at
TBI severity
.90 specificity. Animal fluency validity cutoffs were the only
PVT Scale Cutoff Mild M/S v2 p U2 RR ones that had higher BRFail in patients with moderate/severe
WCT Raw 45 22.4 10.5 1.29 .256 .017 2.13 TBI, consistent with the reports of Curtis et al. (2008) on
VI-7 Raw 3 39.2 12.5 3.94 .047 .047 3.14
Animals T 31 12.1 21.1 0.94 .331 .012 0.57 the FAS. On all other PVT, patients with mild TBI produced
29 10.0 15.8 0.41 .521 .005 0.63 higher BRFail.
RDS Raw 7 24.1 10.5 1.61 .204 .021 2.30 However, other aspects of the findings contradict our
6 12.1 0.0 2.52 .112 .033 NA
Digit Span ACSS 6 29.3 0.0 7.15 .008 .093 NA omnibus prediction. For example, the FAS cutoff T  33
5 19.0 0.0 4.20 .040 .055 NA failed to meet specificity standards in the current sample
Note. TBI: Traumatic brain injury; PVT: Performance validity test; WCT: Word (.75–.84), although it was specific (.92) to non-credible
Choice Test; VI-7: Validity Index Seven; RDS: Reliable Digit Span; ACSS: Age-
corrected scaled score; M/S: Moderate-to-severe; RR: Risk ratio (base rate of responding in the mild TBI subsample in the Curtis et al.
failure among mild TBI patients divided by the base rate of failure among (2008) study. Similarly, in our sample, the FAS cutoff T  31
moderate-to-severe TBI patients). failed to reach minimum specificity against three of the four
criterion PVTs (.81–.83). In contrast, the same cutoff had
RDS cutoff (6) traded sensitivity (.18–.27) for excellent .87 specificity even in patients with moderate-to-severe TBI
specificity (.95–.98). A Digit Span ACSS 6 produced a in the study by Curtis et al. (2008).
good combination of sensitivity (.50–.53) and specificity
(.86–.96). Making the cutoff more conservative (5) dispro-
portionately traded sensitivity (.27–.40) for specifi- Process variables and multivariate models using verbal
city (.92–.96). fluency measures
Despite previous reports that component analysis of verbal
Base rates of PVT failure as a function of injury severity fluency measures provides unique and clinically useful infor-
in the clinical sample mation to characterize the pattern of cognitive deficits in
patients with TBI (Zakzanis, McDonald, & Troyer, 2011;
Given that EVIs are vulnerable to the confluence of genuine 2013), process variables failed to demonstrate a meaningful
cognitive impairment and non-credible responding by psychometric advantage over traditional summary scores in
design (Silverberg et al., 2008), BRFail were compared among predicting non-credible responding. Beyond the total scores,
patients with mild TBI and those with moderate or severe results of logistic regression analyses suggest that there is no
TBI. The mild TBI subsample had a higher BRFail on both statistical benefit to tracking the decline in output over time
the WCT and the VI-7 (RR: 2.13–3.14), although the former for verbal fluency measures as a derivative index of response
contrast failed to reach statistical significance (Table 10). In validity. Similarly, the small gain in classification accuracy
contrast, patients with moderate or severe TBI failed animal hardly justifies the burden of computing and interpreting
fluency validity cutoffs at a higher rate than patients with LRCs that combine the predictive power of animals and
mild TBI (RR: 1.58–1.74, non-significant contrasts). At RDS FAS total scores. Using univariate cutoffs appears to be the
7, BRFail was higher in the mild TBI subsample most parsimonious solution—from a statistical, practical,
(RR ¼ 2.30, non-significant contrast). No patient with mod-
and clinical perspective.
erate or severe TBI failed the RDS 6 cutoff. Similarly, This finding is in direct contradiction with a growing
Digit Span ACSS 6 and 5 had perfect specificity (i.e., a body of evidence on both the advantages of combining mul-
zero BRFail) in the subsample with moderate or severe TBI tiple PVTs to determine the credibility of a neurocognitive
(significant contrasts).
profile in general (Bashem et al., 2014; Davis & Millis, 2014;
Larrabee, 2008, 2014; Larrabee, Rohling, & Meyers, 2019;
Discussion Lichtenstein, Greenacre, et al., 2019; Meyers et al., 2014;
Odland, Lammy, Martin, Grote, & Mittenberg, 2015; Tyson
Review of main findings et al., 2018) and verbal fluency measures specifically
To our knowledge, this is the first study to evaluate the sig- (Johnson et al., 2012; Silverberg et al., 2008). A likely
nal detection performance of validity cutoffs on verbal flu- explanation for this divergence from the existing literature is
ency measures in a non-clinical sample using the expMAL collinearity. Process variables and the total score on the two
paradigm. We predicted that classification accuracy would types of verbal fluency tests (category and letter) contribute
be higher than what was reported in previous research in highly redundant information and thus, fail to enhance the
clinical patients, that verbal fluency based EVIs would be overall predictive power of the more complex model. This
sensitive to TBI severity, and that an inverse dose-response negative finding is a sobering reminder of longstanding
relationship between failure rates on other PVTs and TBI warnings about the importance of using independent PVTs
severity would be observed. Overall, the results supported (i.e., with low inter-correlation) in multivariate models of
these hypotheses. On FAS, sensitivity was consistently higher performance validity assessment (Nelson et al., 2003;
(.40–.54 versus .05–.34) in the student sample with Rosenfeld, Sands, & van Gorp, 2000).
10 J. HURTUBISE ET AL.
Verbal fluency measures as EVIs in a broader context
The effect of expMAL on verbal fluency measures was com-
parable (d: 1.07–1.23) to that on Digit Span variables (d:
1.19–1.26), with the isolated exception of LDF–LDB. This
pattern of findings is consistent with previous research sug-
gesting that derivative validity indices either have inherently
low sensitivity (Abeare, Sabelli, et al., 2019; Arnold et al.,
2005; Axelrod, Fichtenberg, Millis, & Wertheimer, 2006;
Erdodi, Hurtubise, et al., 2018; Erdodi, Sagar, et al., 2018;
Iverson, Lange, Green, & Franzen, 2002; Lichtenstein, Flaro,
Baldwin, Rai, & Erdodi, 2019; Merten, Bossink, & Schmand,
2007; Miller, Ryan, Carruthers, & Cluff, 2004; Powell, Locke,
Smigielski, & McCrea, 2011) or their classification accuracy
may be population-specific (Erdodi, Abeare, et al., 2017;
Glassmire et al., 2019). Verbal fluency and Digit Span based
EVIs were outperformed by free-standing and composite
PVTs (d: 1.25–1.69), re-affirming their superior signal detec-
tion properties.
Consistent with the reports of Whiteside, Kogan, et al.
(2015), animal fluency had higher AUC values (.74–.88)
than FAS (.64–.77). Compared to Digit Span based EVIs
(except LDF–LDB), FAS produced consistently lower AUCs
(.64–.77 versus .75–.90). In contrast, animal fluency achieved
similar AUC values (.74–.88). At the level of individual cut-
offs, animal fluency produced consistently better classifica-
tion accuracy (Table 6). This finding is also consistent with
the results of the two previous studies that provided a direct
comparison between the animal fluency and FAS (Sugarman
& Axelrod, 2015; Whiteside, Kogan, et al., 2015).
Within-Group variability as an index of
performance validity
Reliably higher within-group variability associated with the
expMAL condition was an incidental finding that replicates
previous reports (Kanser, Rapport, Bashem, & Hanks, 2019;
Larrabee et al., 2019; Rapport, Farchione, Coleman, &
Axelrod, 1998). A possible explanation for this is a diver-
gence in malingering strategy: participants may have used
different templates for producing credible impairment
(Cottingham, Victor, Boone, Ziegler, & Zeller, 2014; Erdodi,
Kirsch, Lajiness-O’Neill, Vingilis, & Medoff, 2014). This line
of reasoning underlines the importance of continuous moni-
toring of performance validity (Boone, 2009; Chafetz et al.,
2015; Critchfield et al., 2019; Schutte, Axelrod, & Montoya,
2015) using multiple PVTs that represent a wide range of
cognitive domains, sensory modalities, and testing para-
digms (Boone, 2013; Erdodi, 2019).
Limitations of the experimental malingering paradigm
A more skeptical (and parsimonious) account for the
inflated SDs in the expMAL group is incomplete compre-
hension, lack of experiential basis, or the limited motivation
for the optimal execution of the expMAL instructions (An
et al., 2017). As previous investigators have pointed out,
group assignment is a pseudo-independent variable in
expMAL paradigms: the experimenter only controls the
instructions given to the participants, not the fidelity of their
execution (Rai et al., 2019). Case in point: 28.1% of partici-
pants in the control group failed 1 PVTs, and 14.0% failed
2 PVTs. In other words, a quarter of the sample assumed
to demonstrate valid performance had psychometric evi-
dence to the contrary (Abeare, Messa, et al., 2019; Davis,
Axelrod, McHugh, Hanks, & Millis, 2013; Proto et al.,
2014). While this is consistent with previous reports on fluc-
tuating effort in undergraduate research participants (An
et al., 2017; An, Zakzanis, & Joordens, 2012; Roye et al.,
2019), it provides objective evidence for criterion group con-
tamination. Conversely, 25–40% of the expMAL subsample
scored in the Average range or above on neuropsychological
tests that are sensitive to diffuse neurocognitive deficits
(Bialystok, Craik, Binns, Ossher, & Freedman, 2014; Curtis,
Greve, & Bianchini, 2009; Donders & Strong, 2015; Savla
et al., 2011; Tyson et al., 2018).
Arguably, the discrepancy between the request to produce
credible impairment and the extent to which participants
are willing or able to do so is the Achilles heel of research
designs based on the expMAL paradigm. Anecdotally, some
participants reported struggling with the increased cognitive
demands of simultaneously feigning deficits while avoiding
detection. In defense of participants assigned to the expMAL
condition, their task does appear to be objectively more dif-
ficult. Research on neurophysiological correlates of decep-
tion repeatedly found increased overall activation
(Browndyke et al., 2008; Suchotzki, Crombez, Smulders,
Meijer, & Verschuere, 2015), often outside of the brain
region typically involved in performing the task “to the best
of their ability” (Larsen, Allen, Bigler, Goodrich-Hunsaker,
& Hopkins, 2010), suggesting that (perhaps even unsuccess-
ful) malingering attempts are energetically more expensive
(Yu, Tao, Zhang, Chan, & Lee, 2019).
Is single word reading immune to non-
credible responding?
Another incidental finding was that performance on a sin-
gle-word reading test was robust to the global deleterious
effects of expMAL, replicating earlier reports (Coleman,
Rapport, Millis, Ricker, & Farchione, 1998; Greiffenstein,
Baker, & Gola, 1996; Kanser et al., 2017). In contrast, a
more recent study found a strong linear relationship
between standard scores on a different single-word reading
test and PVT failures (Martin et al., 2018). The same
research group also reported that an excessive decline from
estimated pre-morbid functioning based on performance on
a single-word reading test was an emerging index of non-
credible responding (Martin, Hunter, Rach, Heinrichs, &
Schroeder, 2017).
In a rare example of rigorous experimental control,
Frazier, Frazier, Busch, Kerwood, and Demaree (2008) dem-
onstrated the methodological complexity of isolating the
effect of expMAL. Even the lowest performing group scored
in the Average range. However, this was significantly below
baseline testing (performance obtained outside of expMAL

instructions). This finding suggests that a score within the
normal range does not rule out non-credible responding—
especially in cognitively high functioning examinees, such as
athletes assessed to establish a baseline cognitive functioning
in sports concussion management programs (Abeare, Messa,
Zuccato, Merker, & Erdodi, 2018; Abeare, Messa, et al.,
2019; Gaudet & Weyandt, 2017; Higgins, Denney, &
Maerlender, 2017; Iverson & Schatz, 2015; Lichtenstein,
Linnea, & Maerlender, 2018; McCrea et al., 2003; Tsushima
et al., 2019). Similarly, a large effect emerged as a function
of the type of disorder being feigned. Given the widespread
reliance on such measures to establish baseline cognitive
abilities (Boone, 2013; Green et al., 2008; Johnstone,
Callahan, Kapila, & Bouman, 1996; Mathias, Bowden, Bigler,
& Rosenfeld, 2007; McFarlane, Welch, & Rodgers, 2006;
Steward et al., 2017), more research is clearly needed to
establish the relationship between single-word reading tests
and invalid performance.
The attenuation of classification accuracy from student
to clinical sample
Naturally, correctly identifying invalid performance presents
a greater psychometric challenge in a clinical sample where
genuine deficits and non-credible responding can coexist.
Nevertheless, only a marginal shrinkage in specificity was
observed on animal fluency cutoffs from students to patients
at T  31 (from .86–.93 to .89–.91) and T  29 (from .94–.98
to .94–.96). The loss of signal detection power was more
apparent at RDS 7 (from .93–.97 among students to
.86–.89 among patients with TBI). Although specificity val-
ues were comparable at RDS 6 between the two samples
(.98–1.00 vs .95–.98), sensitivity was notably higher in the
student sample (.38–.53 vs .18–.27). Finally, Digit Span
ACSS 6 had produced similar combinations of sensitivity
and specificity in both samples, although classification
accuracy was consistently higher among students. In the
clinical sample, Digit Span ACSS had better classification
accuracy than RDS. This finding is consistent with most
recent reports (Shura, Martindale, Taber, Higgins, &
Rowland, 2019) and the cumulative evidence (Babikian,
Boone, Lu, & Arnold, 2006; Kiewel, Wisdom, Bradshaw,
Pastorek, & Strutt, 2012; Spencer et al., 2013; Webber &
Soble, 2018; Whitney et al., 2009).
Strengths and limitations of the study
As the first investigation of verbal fluency measures as EVIs
in a non-clinical sample, this study extended the scope of
these instruments to cognitively intact populations. Previous
research suggests that since most PVTs were designed to be
robust to genuine deficits, their utility is limited in high
functioning examinees (Abeare, Messa, et al., 2019; An
et al., 2017; Roye et al., 2019). Animal and letter fluency cut-
offs were validated against a variety of criterion PVTs, pro-
viding a thorough test of their signal detection properties.
Results support the need for population-specific cutoffs.
APPLIED NEUROPSYCHOLOGY: CHILD 11
Inevitably, the study has several limitations, too. It was
based on a sample of convenience recruited from a single
location. Thus, results may not generalize to other geo-
graphic regions (Kura, 2013; Le on & Le on, 2014;
Lichtenstein, Greenacre, et al., 2019; Lynn, 2010; McDaniel,
2006; Roth, Erdodi, McCulloch, & Isquith, 2015). In add-
ition, participants were limited to healthy young adults.
Since age has been reported to influence base rates of PVT
failure, especially in children (Abeare et al., 2018;
Lichtenstein, Holcomb, & Erdodi, 2018), future studies may
benefit from expanding the age range of the examinees. The
choice of criterion measures may have also inadvertently
influenced the classification accuracy (Erdodi, 2019; Rai &
Erdodi, 2019; Schroeder et al., 2019; Schwartz et al., 2016).
Replication is needed using different samples and criterion
PVTs. Likewise, involving newer versions of verbal fluency
measures such as the Action (Piatt, Fields, Paolo, & Troster,
1999) and Emotion Word Fluency Test (Abeare et al., 2017)
could further enhance the expanding knowledge base on
this family of EVIs. Finally, as described above, the expMAL
paradigm itself has a number of inherent epistemological
limitations.
As the student sample consisted of healthy young adults
from a single location, the extent to which results would
generalize to older clinical patients in different geographic
regions remains an open question. At the same time, even
the control group scored significantly below the normative
mean on both measures of verbal fluency—a puzzling find-
ing that is not without precedent (Abeare, Messa, et al.,
2019; An et al., 2017). Equally surprisingly, the FAS cutoff
(T  33) that was specific (.92) to non-credible responding
in patients with mild TBI in the Curtis et al. (2008) study
failed to clear the minimum specificity threshold in the pre-
sent sample of cognitively intact participants. These findings
suggest that the commonly used clinical vs. non-clinical dis-
tinction may be an artificial dichotomy. At least in terms of
verbal fluency scores, there seems to be a substantial overlap
in mean performance between healthy controls (Abeare,
Messa, et al., 2019; An et al., 2017; Erdodi, Jongsma, & Issa,
2017; McCrea et al., 2003; Piatt, Fields, Paolo, Koller, &
Troster, 1999) and clinical patients (Johnson et al., 2012;
Piatt, Fields, Paolo, Koller, et al., 1999; Sugarman &
Axelrod, 2015; Tyson et al., 2018; Whiteside, Kogan, et al.,
2015; Zakzanis et al., 2011, 2013).
Nevertheless, given the sensitivity of verbal fluency meas-
ures to diffuse neuropsychiatric disorders (Henry &
Crawford, 2004; Iverson, Franzen, & Lovell, 1999; Loring
et al., 1995), low scores on these tests should not be inter-
preted as evidence of non-credible responding until alterna-
tive explanations (i.e., genuine deficits due to various
etiologies ranging from neurodevelopmental disorders
through acquired aphasia to limited English proficiency)
have been ruled out (Bodner et al., 2019; Boone, Victor,
Wen, Razani, & Pont on, 2007; Erdodi, Nussbaum, Sagar,
Abeare, & Schwartz, 2017; Schroeder & Marshall, 2010).
Consequently, although failing validity cutoffs on FAS or
animals can be helpful in ruling in invalid performance,
they should never be used in isolation to determine the
12 J. HURTUBISE ET AL.
credibility of the overall neurocognitive profile. However, in
combination with failures on robust free-standing PVTs,
they can provide incremental evidence to improve the asses-
sor’s confidence in the final conclusion (Erdodi, 2019;
Lippa, 2018; Merten & Rogers, 2017; Odland et al., 2015).
Future studies that provide a direct comparison between
healthy controls, experimental malingerers and credible
patients with genuine deficits could further illuminate the
potential confluence of legitimate impairment and invalid
performance. Such a design would allow researchers to esti-
mate the relative contribution of credible deficits and non-
credible responding (Giromini, Viglione, Pignolo, &
Zennaro, 2018; Hopwood, Morey, Rogers, & Sewell, 2007;
Kanser et al., 2019). Including an additional group of
patients with medically verified conditions known to be
associated with decreased verbal fluency performance (who
are also instructed to malinger) would further enhance the
heuristic value of future research designs (Giromini et al.,
2019; Inman & Berry, 2002; Iverson, Franzen, & Mccracken,
1994; Rogers et al., 2003; Rose, Hall, Szalda-Petree, & Bach,
1998; Vickery et al., 2004; Viglione et al., 2019; von
Helvoort, Merckelbach, & Merten, 2019).
Conclusions
Verbal fluency measures were effective at separating valid
from invalid response sets in a non-clinical sample of under-
graduate student volunteers using the expMAL paradigm.
Surprisingly, FAS cutoffs previously validated in clinical
patients (Curtis et al., 2008) failed to achieve minimum spe-
cificity standards in the present sample, underlining the
importance of population-specific cutoffs. Consistent with
previous research, animal fluency had higher classification
accuracy than FAS. However, both EVIs performed notably
better in the student sample than in previous studies and
had a comparable signal detection profile to Digit Span vari-
ables as EVIs. The most plausible explanation to this pattern
of finding is that, as a test of cognitive ability, verbal fluency
is more sensitive to diffuse neuropsychological deficits than
Digit Span.
As such, genuine impairment and non-credible respond-
ing may be more closely intertwined in clinical patients and,
thus, more difficult to differentiate using verbal fluency
measures. Therefore, failing validity cutoffs on the FAS or
animals should not be interpreted as conclusive evidence of
invalid performance. Indeed, patients with moderate or
severe TBI had a higher BRFail on animal fluency validity
cutoffs, while they had lower BRFail on all other PVTs exam-
ined. Nevertheless, before automatically discounting an EVI
failure as a false positive, the broader clinical context should
be considered, in conjunction with the ongoing consultation
of the empirical literature. In fact, on the EVIs examined in
this study, conservative cutoffs provided an effective safe-
guard against false positive errors even among patients with
verified neurological disorders. Ultimately, the golden rule
remains that these EVIs should only be used and interpreted
in combination with other PVTs that are known to be
robust to genuine impairment.
Ethical approval
Relevant ethical guidelines were followed throughout the
project. All data collection, storage and processing was done
with the approval of relevant institutional authorities regu-
lating research involving human participants, in compliance
with the 1964 Helsinki Declaration and its subsequent
amendments or comparable ethical standards.
Disclosure statement
The last author provides forensic consultation and medicolegal assess-
ments, for which he receives financial compensation.
References
Abeare, C. A., Freund, S., Kaploun, K., McAuley, T., & Dumitrescu, C.
(2017). The Emotion Word Fluency Test (EWFT): Initial psycho-
metric, validation, and physiological evidence in young adults.
Journal of Clinical and Experimental Neuropsychology, 39(8),
738–752. doi:10.1080/13803395.2016.1259396
Abeare, C., Messa, I., Whitfield, C., Zuccato, B., Casey, J., Rykulski, N.,
& Erdodi, L. (2019). Performance validity in collegiate football ath-
letes at baseline neurocognitive testing. Journal of Head Trauma
Rehabilitation, 34(4), E20–E31. doi:10.1097/HTR.0000000000000451
Abeare, C. A., Messa, I., Zuccato, B. G., Merker, B., & Erdodi, L. A.
(2018). Prevalence of invalid performance on baseline testing for
sport-related concussion by age and validity indicator. JAMA
Neurology, 75(6), 697–703. doi:10.1001/jamaneurol.2018.0031
Abeare, C., Sabelli, A., Taylor, B., Holcomb, M., Dumitrescu, C.,
Kirsch, N., & Erdodi, L. (2019). The importance of demographically
adjusted cutoffs: Age and education bias in raw score cutoffs within
the Trail Making Test. Psychological Injury and Law, 12(2), 170–182.
doi:10.1007/s12207-019-09353
An, K. Y., Charles, J., Ali, S., Enache, A., Dhuga, J., & Erdodi, L. A.
(2019). Re-examining performance validity cutoffs within the
Complex Ideational Material and the Boston Naming Test-Short
Form using an experimental malingering paradigm. Journal of
Clinical and Experimental Neuropsychology, 41(1), 15–25. doi:10.
1080/13803395.2018.1483488
An, K. Y., Kaploun, K., Erdodi, L. A., & Abeare, C. A. (2017).
Performance validity in undergraduate research participants: A com-
parison of failure rates across tests and cutoffs. The Clinical
Neuropsychologist, 31(1), 193–206. doi:10.1080/13854046.2016.
1217046
An, K. Y., Zakzanis, K. K., & Joordens, S. (2012). Conducting research
with non-clinical healthy undergraduates: Does effort play a role in
neuropsychological test performance? Archives of Clinical
Neuropsychology, 27(8), 849–857. doi:10.1093/arclin/acs085
Arnold, G., Boone, K. B., Lu, P., Dean, A., Wen, J., Nitch, S., &
McPherson, S. (2005). Sensitivity and specificity of finger tapping
test scores for the detection of suspect effort. The Clinical
Neuropsychologist, 19(1), 105–120. doi:10.1080/13854040490888567
Ashendorf, L., Clark, E. L., & Sugarman, M. A. (2017). Performance
validity and processing speed in a VA polytrauma sample. The
Clinical Neuropsychologist, 31(5), 857–866. doi:10.1080/13854046.
2017.1285961
Axelrod, B. N., Fichtenberg, N. L., Millis, S. R., & Wertheimer, J. C.
(2006). Detecting incomplete effort with Digit Span from the
Wechsler Adult Intelligence Scale – Third Edition. The Clinical
Neuropsychologist, 20(3), 513–523. doi:10.1080/13854040590967117
Axelrod, B. N., Meyers, J. E., & Davis, J. J. (2014). Finger tapping test
performance as a measure of performance validity. The Clinical
Neuropsychologist, 28(5), 876–888. doi:10.1080/13854046.2014.907583
Babikian, T., Boone, K. B., Lu, P., & Arnold, G. (2006). Sensitivity and
specificity of various Digit Span scores in the detection of suspect

effort. The Clinical Neuropsychologist, 20(1), 145–159. doi:10.1080/
13854040590947362
Backhaus, S. L., Fichtenberg, N. L., & Hanks, R. A. (2004). Detection
of sub-optimal performance using a floor effect strategy in patients
with traumatic brain injury. The Clinical Neuropsychologist, 18(4),
591–603. doi:10.1080/13854040490888558
Bailey, K. C., Soble, J. R., Bain, K. M., & Fullen, C. (2018). Embedded
performance validity tests in the Hopkins Verbal Learning Test-
Revised and the Brief Visuospatial Memory Test-Revised: A replica-
tion study. Archives of Clinical Neuropsychology, 33(7), 895–900. doi:
10.1093/arclin/acx111
Bain, K. M., & Soble, J. R. (2019). Validation of the Advanced Clinical
Solutions Word Choice Test (WCT) in a mixed clinical sample:
Establishing classification accuracy, sensitivity/specificity, and cutoff
scores. Assessment, 26(7), 1320–1328. doi:10.1177/1073191117725172
Barhon, L. I., Batchelor, J., Meares, S., Chekaluk, E., & Shores, E. A.
(2015). A comparison of the degree of effort involved in the TOMM
and the ACS Word Choice Test using a dual-task paradigm. Applied
Neuropsychology: Adult, 22(2), 114–123.
Bashem, J. R., Rapport, L. J., Miller, J. B., Hanks, R. A., Axelrod, B. N.,
& Millis, S. R. (2014). Comparison of five performance validity indi-
ces in bona fide and simulated traumatic brain injury. The Clinical
Neuropsychologist, 28(5), 851–875. doi:10.1080/13854046.2014.927927
Berger, C., Lev, A., Braw, Y., Elbaum, T., Wagner, M., & Rassovsky, Y.
(2019). Detection of feigned ADHD using the MOXO-d-CPT.
Journal of Attention Disorders. doi:10.1177/1087054719864656
Bialystok, E., Craik, F. I. M., Binns, M. A., Ossher, L., & Freedman, L.
(2014). Effects of bilingualism on the age of onset and progression
of MCI and AD: Evidence from executive function tests.
Neuropsychology, 28(2), 290–304. doi:10.1037/neu0000023
Bigler, E. D. (2012). Symptom validity testing, effort and neuropsycho-
logical assessment. Journal of the International Neuropsychological
Society, 18(4), 632–642. doi:10.1017/S1355617712000252
Bigler, E. D. (2014). Effort, symptom validity testing, performance val-
idity testing and traumatic brain injury. Brain Injury, 28(13–14),
1623–1638. doi:10.3109/02699052.2014.947627
Bigler, E. D. (2015). Neuroimaging as a biomarker in symptom validity
and performance validity testing. Brain Imaging and Behavior, 9(3),
421–444. doi:10.1007/s11682-015-9409-1
Bodner, T., Merten, T., & Benke, T. (2019). Performance validity meas-
ures in clinical patients with aphasia. Journal of Clinical and
Experimental Neuropsychology, 41(5), 476–483. doi:10.1080/
13803395.2019.1579783
Boone, K. B. (2013). Clinical practice of forensic neuropsychology. New
York, NY: Guilford.
Boone, K. B. (2009). The need for continuous and comprehensive sam-
pling of effort/response bias during neuropsychological examination.
The Clinical Neuropsychologist, 23(4), 729–741. doi:10.1080/
13854040802427803
Boone, K. B., Salazar, X., Lu, P., Warner-Chacon, K., & Razani, J.
(2002). The Rey 15-item recognition trial: A technique to enhance
sensitivity of the Rey 15-item memorization test. Journal of Clinical
and Experimental Neuropsychology, 24(5), 561–573. doi:10.1076/jcen.
24.5.561.1004
Boone, K. B., Victor, T. L., Wen, J., Razani, J., & Ponton, M. (2007).
The association between neuropsychological scores and ethnicity,
language, and acculturation variables in a large patient population.
Archives of Clinical Neuropsychology, 22(3), 355–365. doi:10.1016/j.
acn.2007.01.010
Bortnik, K. E., Boone, K. B., Marion, S. D., Amano, S., Ziegler, E.,
Victor, T. L., & Zeller, M. A. (2010). Examination of various WMS-
III logical memory scores in the assessment of response bias. The
Clinical Neuropsychologist, 24(2), 344–357. doi:10.1080/
13854040903307268
Boskovic, I., Biermans, A. J., Merten, T., Jelicic, M., Hope, L., &
Merckelbach, H. (2018). The Modified Stroop Task is susceptible to
feigning: Stroop performance and symptom over-endorsement in
feigned test anxiety. Frontiers in Psychology, 9, 1195. doi:10.3389/
fpsyg.2018.01195
APPLIED NEUROPSYCHOLOGY: CHILD 13
Brandt, J., & Benedict, R. H. B. (2001). Hopkins Verbal Learning Test –
Revised. Odessa, FL: Psychological Assessment Services.
Brennan, A. M., Meyer, S., David, E., Pella, R., Hill, B. D., & Gouvier,
W. D. (2009). The vulnerability to coaching across measures of
effort. The Clinical Neuropsychologist, 23(2), 314–328. doi:10.1080/
13854040802054151
Browndyke, J. N., Paskavitz, J., Sweet, L. H., Cohen, R. A., Tucker,
K. A., Welsh-Bohmer, K. A., … Schmechel, D. E. (2008).
Neuroanatomical correlates of malingered memory impairment:
Event-related fMRI of deception on a recognition memory task.
Brain Injury, 22(6), 481–489. doi:10.1080/02699050802084
Bush, S. S., Heilbronner, R. L., & Ruff, R. M. (2014). Psychological
assessment of symptom and performance validity, response bias,
and malingering: Official position of the Association for Scientific
Advancement in Psychological Injury and Law. Psychological Injury
and Law, 7(3), 197–205. doi:10.1007/s12207-014-9198-7
Bush, S., Ruff, R., Troster, A., Barth, J., Koffler, S., Pliskin, N., …
Silver, C. (2005). Symptom validity assessment: Practice issues and
medical necessity (NAN Policy and Planning Committees). Archives
of Clinical Neuropsychology, 20(4), 419–426. doi:10.1016/j.acn.2005.
02.002
Carone, D. A., Green, P., & Drane, D. L. (2014). Word memory test
profiles in two cases with surgical removal of the left anterior hippo-
campus and parahippocampal gyrus. Applied Neuropsychology:
Adult, 21(2), 155–160. doi:10.1080/09084282.2012.755533
Chafetz, M. D., Williams, M. A., Ben-Porath, Y. S., Bianchini, K. J.,
Boone, K. B., Kirkwood, M. W., … Ord, J. S. (2015). Official pos-
ition of the American Academy of Clinical Neuropsychology Social
Security Administration policy on validity testing: Guidance and rec-
ommendations for change. The Clinical Neuropsychologist, 29(6),
723–740. doi:10.1080/13854046.2015.1099738
Coleman, R. D., Rapport, L. J., Millis, S. R., Ricker, J. H., & Farchione,
T. J. (1998). Effects of coaching on detection of malingering on the
California Verbal Learning Test. Journal of Clinical and
Experimental Neuropsychology, 20(2), 201–210. doi:10.1076/jcen.20.2.
201.1164
Cottingham, M. E., Victor, T. L., Boone, K. B., Ziegler, E. A., & Zeller,
M. (2014). Apparent effect of type of compensation seeking (disabil-
ity vs. litigation) on performance validity test scores may be due to
other factors. The Clinical Neuropsychologist, 28(6), 1030–1047. doi:
10.1080/13854046.2014.951397
Critchfield, E., Soble, J. R., Marceaux, J. C., Bain, K. M., Chase Bailey,
K., Webber, T. A. … O’Rourke, J. J. F. (2019). Cognitive impair-
ment does not cause invalid performance: Analyzing performance
patterns among cognitively unimpaired, impaired, and noncredible
participants across six performance validity tests. The Clinical
Neuropsychologist, 33(6), 1083–1101. doi:10.1080/13854046.2018.
1508615
Crowe, S. F. (1996). Traumatic anosmia coincides with an organic dis-
inhibition syndrome as measured by responding on the Controlled
Oral Word Association Test. Psychiatry, Psychology and Law, 3(1),
39–45. doi:10.1080/13218719609524873
Curtis, K. L., Greve, K. W., & Bianchini, K. J. (2009). The Wechsler
Adult Intelligence Scale-III and malingering in traumatic brain
injury. Assessment, 16(4), 401–414. doi:10.1177/1073191109338161
Curtis, K. L., Thompson, L. K., Greve, K. W., & Bianchini, K. J. (2008).
Verbal fluency indicators of malingering in traumatic brain injury:
Classification accuracy in known groups. The Clinical
Neuropsychologist, 22(5), 930–945. doi:10.1080/13854040701563591
Dandachi-FitzGerald, B., Merckelbach, H., & Ponds, R. W. H. M.
(2017). Neuropsychologists’ ability to predict distorted symptom
presentation. Journal of Clinical and Experimental Neuropsychology,
39(3), 257–264. doi:10.1080/13803395.2016.1223278
Davis, J. J. (2014). Further consideration of Advanced Clinical
Solutions Word Choice: Comparison to the Recognition Memory
Test – Words and classification accuracy on a clinical sample. The
Clinical Neuropsychologist, 28(8), 1278–1294. doi:10.1080/13854046.
2014.975844
Davis, J. J., Axelrod, B. N., McHugh, T. S., Hanks, R. A., & Millis, S. R.
(2013). Number of impaired scores as a performance validity
14 J. HURTUBISE ET AL.
indicator. Journal of Clinical and Experimental Neuropsychology,
35(4), 413–420. doi:10.1080/13803395.2013.781134
Davis, J. J., & Millis, S. R. (2014). Examination of performance validity
test failure in relation to number of tests administered. The Clinical
Neuropsychologist, 28(2), 199–214. doi:10.1080/13854046.2014.884633
Delis, D. C., Kaplan, E. F., & Kramer, J. H. (2001). Delis-Kaplan execu-
tive function system. San Antonio, TX: Psychological Corporation.
DenBoer, J. W., & Hall, S. (2007). Neuropsychological test performance
of successful brain injury simulators. The Clinical Neuropsychologist,
21(6), 943–955. doi:10.1080/13854040601020783
Donders, J., & Strong, C. A. H. (2015). Clinical utility of the Wechsler
Adult Intelligence Scale – Fourth Edition after traumatic brain
injury. Assessment, 22(1), 17–22. doi:10.1177/1073191114530776
Eglit, G. M. L., Jurick, S. M., Delis, D. C., Filoteo, J. V., Bondi, M. W.,
& Jak, A. J. (2019). Utility of the D-KEFS Color Word Interference
Test as an embedded measure of performance validity. The Clinical
Neuropsychologist, 1–21. doi:10.1080/13854046.2019.1643923
Erdodi, L. A., & Abeare, C. A. (2019). Stronger together: The Wechsler
Adult Intelligence Scale – Fourth Edition as a multivariate perform-
ance validity test in patients with traumatic brain injury. Archives of
Clinical Neuropsychology. doi:10.1093/arclin/acz032/5613200
Erdodi, L. A., Abeare, C. A., Medoff, B., Seke, K. R., Sagar, S., &
Kirsch, N. L. (2018). A single error is one too many: The Forced
Choice Recognition trial on the CVLT-II as a measure of perform-
ance validity in adults with TBI. Archives of Clinical
Neuropsychology, 33(7), 845–860. doi:10.1093/arclin/acx110
Erdodi, L. A., Roth, R. M., Kirsch, N. L., Lajiness-O’Neill, R., &
Medoff, B. (2014). Aggregating validity indicators embedded in
Conners’ CPT-II outperforms individual cutoffs at separating valid
from invalid performance in adults with traumatic brain injury.
Archives of Clinical Neuropsychology, 29(5), 456–466. doi:10.1093/
arclin/acu026
Erdodi, L. A., Taylor, B., Sabelli, A. G., Malleck, M., Kirsch, N. L., &
Abeare, C. A. (2019). Demographically adjusted validity cutoffs on
the Finger Tapping Test are superior to raw score cutoffs in adults
with TBI. Psychological Injury and Law, 12(2), 113–126. doi:10.1007/
s12207-019-09352-y
Erdodi, L. A. (2019). Aggregating validity indicators: The salience of
domain specificity and the indeterminate range in multivariate mod-
els of performance validity assessment. Applied Neuropsychology:
Adult, 26(2), 155–172. doi:10.1080/23279095.2017.1384925
Erdodi, L. A., Abeare, C. A., Lichtenstein, J. D., Tyson, B. T.,
Kucharski, B., Zuccato, B. G., & Roth, R. M. (2017). WAIS-IV proc-
essing speed scores as measures of non-credible responding – The
third generation of embedded performance validity indicators.
Psychological Assessment, 29(2), 148–157. doi:10.1037/pas0000319
Erdodi, L. A., Dunn, A. G., Seke, K. R., Charron, C., McDermott, A.,
Enache, A., … Hurtubise, J. (2018). The Boston Naming Test as a
measure of performance validity. Psychological Injury and Law,
11(1), 1–8. doi:10.1007/s12207-017-9309-3
Erdodi, L. A., Green, P., Sirianni, C., & Abeare, C. A. (2019). The
myth of high false positive rates on the Word Memory Test in mild
TBI. Psychological Injury and Law, 12(2), 155–169. doi:10.1007/
s12207-019-09356-8
Erdodi, L. A., Hurtubise, J. L., Charron, C., Dunn, A., Enache, A.,
McDermott, A., & Hirst, R. (2018). The D-KEFS Trails as perform-
ance validity tests. Psychological Assessment, 30(8), 1082–1095. doi:
10.1037/pas0000561
Erdodi, L. A., Jongsma, K. A., & Issa, M. (2017). The 15-item version
of the Boston Naming test as an index of English proficiency. The
Clinical Neuropsychologist, 31(1), 168–178. doi:10.1080/13854046.
2016.1224392
Erdodi, L. A., Kirsch, N. L., Lajiness-O’Neill, R., Vingilis, E., & Medoff,
B. (2014). Comparing the Recognition Memory Test and the Word
Choice Test in a mixed clinical sample: Are they equivalent?
Psychological Injury and Law, 7(3), 255–263.
Erdodi, L. A., Kirsch, N. L., Sabelli, A. G., & Abeare, C. A. (2018). The
Grooved Pegboard Test as a validity indicator – A study on psycho-
genic interference as a confound in performance validity research.
Psychological Injury and Law, 11(4), 307–324. doi:10.1007/s12207-
018-9337-7
Erdodi, L. A., & Lichtenstein, J. D. (2019). Information processing
speed tests as PVTs. In K. B. Boone (Ed.), Assessment of feigned cog-
nitive impairment. A neuropsychological perspective. New York, NY:
Guilford.
Erdodi, L. A., & Lichtenstein, J. D. (2017). Invalid before impaired: An
emerging paradox of embedded validity indicators. The Clinical
Neuropsychologist, 31(6–7), 1029–1046. doi:10.1080/13854046.2017.
1323119
Erdodi, L. A., Nussbaum, S., Sagar, S., Abeare, C. A., & Schwartz, E. S.
(2017). Limited English proficiency increases failure rates on per-
formance validity tests with high verbal mediation. Psychological
Injury and Law, 10(1), 96–103. doi:10.1007/s12207-017-9282-x
Erdodi, L. A., & Rai, J. K. (2017). A single error is one too many:
Examining alternative cutoffs on Trial 2 on the TOMM. Brain
Injury, 31(10), 1362–1368. doi:10.1080/02699052.2017.1332386
Erdodi, L. A., Pelletier, C. L., & Roth, R. M. (2018). Elevations on
select Conners’ CPT-II scales indicate noncredible responding in
adults with traumatic brain injury. Applied Neuropsychology: Adult,
25(1), 19–28. doi:10.1080/23279095.2016.1232262
Erdodi, L. A., Sagar, S., Seke, K., Zuccato, B. G., Schwartz, E. S., &
Roth, R. M. (2018). The Stroop Test as a measure of performance
validity in adults clinically referred for neuropsychological assess-
ment. Psychological Assessment, 30(6), 755–766. doi:10.1037/
pas0000525
Erdodi, L. A., Seke, K. R., Shahein, A., Tyson, B. T., Sagar, S., & Roth,
R. M. (2017). Low scores on the Grooved Pegboard Test are associ-
ated with invalid responding and psychiatric symptoms. Psychology
& Neuroscience, 10(3), 325–344. doi:10.1037/pne0000103
Erdodi, L. A., Tyson, B. T., Abeare, C. A., Lichtenstein, J. D., Pelletier,
C. L., Rai, J. K., & Roth, R. M. (2016). The BDAE Complex
Ideational Material – A measure of receptive language or perform-
ance validity?. Psychological Injury and Law, 9(2), 112–120. doi:10.
1007/s12207-016-9254-6
Erdodi, L. A., Tyson, B. T., Abeare, C. A., Zuccato, B. G., Rai, J. K.,
Seke, K. R., … Roth, R. M. (2018). Utility of critical items within
the Recognition Memory Test and Word Choice Test. Applied
Neuropsychology: Adult, 25(4), 327–339. doi:10.1080/23279095.2017.
1298600
Erdodi, L. A., Tyson, B. T., Shahein, A., Lichtenstein, J. D., Abeare,
C. A., Pelletier, C. L., … Roth, R. M. (2017). The power of timing:
Adding a time-to-completion cutoff to the Word Choice Test and
Recognition Memory Test improves classification accuracy. Journal
of Clinical and Experimental Neuropsychology, 39(4), 369–383. doi:
10.1080/13803395.2016.1230181
Erdodi, L. A., & Roth, R. M. (2017). Low scores on BDAE Complex
Ideational Material are associated with invalid performance in adults
without aphasia. Applied Neuropsychology: Adult, 24(3), 264–274.
doi:10.1080/23279095.2017.1298600
Frazier, T., Frazier, A., Busch, R., Kerwood, M., & Demaree, H. (2008).
Detection of simulated ADHD and reading disorder using symptom
validity measures. Archives of Clinical Neuropsychology, 23(5),
501–509.
Fuermaier, A. B. M., Tucha, O., Koerts, J., Grabski, M., Lange, K. W.,
Weisbrod, M., … Tucha, L. (2016). The development of an
embedded figures test for the detection of feigned Attention Deficit/
Hyperactivity Disorder in Adulthood. PLoS One, 11(10), e0164297.
doi:10.1037/journal.pone.0164297
Fuermaier, A. B. M., Tucha, O., Koerts, J., Lange, K. W., Weisbrod,
M., Aschenbrenner, S., & Tucha, L. (2017). Noncredible cognitive
performance at clinical evaluation of adult ADHD: An embedded
validity indicator in a visuospatial working memory test.
Psychological Assessment, 29(12), 1466–1479. doi:10.1037/pas0000534
Gaudet, C. E., & Weyandt, L. L. (2017). Immediate Post-Concussion
and Cognitive Testing (ImPACT): A systematic review of the preva-
lence and assessment of invalid performance. The Clinical
Neuropsychologist, 31(1), 43–58. doi:10.1080/13854046.2016.1220622
Giromini, L., Viglione, D. J., Pignolo, C., & Zennaro, A. (2018). A clin-
ical comparison, simulation study testing the validity and IOP-29

with an Italian sample. Psychological Injury and Law, 11(4),
340–350. doi:10.1007/s12207-018-9314-1
Giromini, L., Lettieri, S. C., Zizolfi, S., Zizolfi, D., Viglione, D.J.,
Brusadelli, E., … Zennaro, A. (2019). Beyond rare-symptom
endorsement: A clinical comparison study using the Minnesota
Multiphasic Personality Inventory-2 (MMPI-2) with the Inventory
of Problems-29 (IOP-29). Psychological Injury and Law, 12(3–4),
212–224. doi:10.1007/s12207-019-09357-7
Gladsjo, J. A., Schuman, C. C., Evans, J. D., Peavy, G. M., Miller,
S. W., & Heaton, R. K. (1999). Norms for letter and category flu-
ency: Demographic corrections for age, education, and ethnicity.
Assessment, 6(2), 147–178. doi:10.1177/107319119900600204
Glassmire, D. M., Wood, M. E., Ta, M. T., Kinney, D. I., & Nitch, S. R.
(2019). Examining false-positive rates of Wechsler Adult Intelligence
Scale (WAIS-IV) processing speed based embedded validity indica-
tors among individuals with schizophrenia spectrum disorders.
Psychological Assessment, 31(1), 120–125.
Goodglass, H., Kaplan, E., & Barresi, B. (2001). Boston Diagnostic
Aphasia Examination (3rd ed.). Philadelphia: Lippincott Williams &
Wilkins.
Green, P., Montijo, J., & Brockhaus, R. (2011). High specificity of the
Word Memory Test and Medical Symptom Validity Test in groups
with severe verbal memory impairment. Applied Neuropsychology,
18(2), 86–94. doi:10.1080/09084282.2010.523389
Green, R. E. A., Melo, B., Christensen, B., Ngo, L. A., Monette, G., &
Bradbury, C. (2008). Measuring premorbid IQ in traumatic brain
injury: An examination of the validity of the Wechsler Test of Adult
Reading (WTAR). analysis. Journal of Clinical and Experimental
Neuropsychology, 30(2), 163–172. doi:10.1080/13803390701300524
Greiffenstein, M. F., Baker, W. J., & Gola, T. (1996). Motor dysfunction
profiles in traumatic brain injury and postconcussion syndrome.
Journal of the International Neuropsychological Society, 2(6),
477–485. doi:10.1017/S1355617700001648
Greve, K. W., Bianchini, K. J., Mathias, C. W., Houston, R. J., &
Crouch, J. A. (2002). Detecting malingered neurocognitive dysfunc-
tion with the Wisconsin Card Sorting Test: A preliminary investiga-
tion in traumatic brain injury. The Clinical Neuropsychologist, 16(2),
179–191. doi:10.1076/clin.16.2.179.13241
Hayward, L., Hall, W., Hunt, M., & Zubrick, S. R. (1987). Can localized
brain impairment be simulated on neuropsychological test profiles?.
Australian & New Zealand Journal of Psychiatry, 21(1), 87–93. doi:
10.3109/00048678709160904
Heaton, R. K., Miller, S. W., Taylor, M. J., & Grant, I. (2004). Revised
comprehensive norms for an expanded Halstead-Reitan battery:
Demographically adjusted neuropsychological norms for African
American and Caucasian adults. Lutz, FL: Psychological Assessment
Resources.
Heaton, R. K., Smith, H. H., Lehman, R. A. W., & Vogt, A. T. (1978).
Prospects for faking believable deficits on neuropsychological test-
ing. Journal of Consulting and Clinical Psychology, 46(5), 892–900.
doi:10.1037/0022-006X.46.5.892
Heilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. J., Millis,
S. R., & Conference, P. (2009). American Academy of Clinical
Neuropsychology consensus conference statement on the neuro-
psychological assessment of effort, response bias, and malingering.
The Clinical Neuropsychologist, 23, 1093–1129. doi:10.1080/
13854040903155063
Henry, J. D., & Crawford, J. R. (2004). A meta-analytic review of verbal
fluency performance following focal cortical lesions.
Neuropsychology, 18(2), 284–295. doi:10.1037/0894-4105.18.2.284
Higgins, K. L., Denney, R. L., & Maerlender, A. (2017). Sandbagging
on the Immediate Post-Concussion Assessment and Cognitive
Testing (ImPACT) in a high school athlete population. Archives of
Clinical Neuropsychology, 32(3), 259–266. doi:10.1093/arclin/acw108
Hilsabeck, R. C. (2017). Psychometrics and statistics: Two pillars of
neuropsychological practice. The Clinical Neuropsychologist, 31(6–7),
995–999. doi:10.1080/13854046.2017.1350752
Hopwood, C., Morey, L. C., Rogers, R., & Sewell, K. (2007).
Malingering on the Personality Assessment Inventory: Identification
APPLIED NEUROPSYCHOLOGY: CHILD 15
of specific feigned disorders. Journal of Personality Assessment,
88(1), 43–48. doi:10.1080/00223890709336833
Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression (2nd
ed.). New York: Wiley.
Inman, T. H., & Berry, D. T. R. (2002). Cross-validation of indicators
of malingering: A comparison of nine neuropsychological tests, four
tests of malingering, and behavioral observations. Archives of
Clinical Neuropsychology, 17(1), 1–23. doi:10.1016/S0887-
6177(00)00073-1
Iverson, G. L., Franzen, M. D., & Lovell, M. R. (1999). Normative com-
parisons for the Controlled Oral Word Association Test following
acute traumatic brain injury. The Clinical Neuropsychologist, 13(4),
437–441. doi:10.1076/1385-4046(199911)13:04;1-Y;FT437
Iverson, G. L., Franzen, M. D., & Mccracken, L. M. (1994). Application
of a forced-choice memory procedure designed to detect experimen-
tal malingering. Archives of Clinical Neuropsychology, 9(5), 437–450.
doi:10.1093/arclin/9.5.437
Iverson, G. L., Lange, R. T., Green, P., & Franzen, M. D. (2002).
Detecting exaggeration and malingering with the Trail Making Test.
The Clinical Neuropsychologist, 16(3), 398–406. doi:10.1076/clin.16.3.
398.13861
Iverson, G. L., & Schatz, P. (2015). Advanced topics in neuropsycho-
logical assessment following sports-related concussion. Brain Injury,
29(2), 263–275. doi:10.3109/02699052.2014.965214
Johnson, S. C., Silverberg, N. D., Millis, S. R., & Hanks, R. A. (2012).
Symptom validity indicators embedded in the Controlled Oral Word
Association Test. The Clinical Neuropsychologist, 26(7), 1230–1241.
doi:10.1080/13854046.2012.709886
Johnstone, B., Callahan, C. D., Kapila, C. J., & Bouman, D. E. (1996).
The comparability of the WRAT-R Reading test and the NAART as
estimates of premorbid intelligence in neurologically impaired
patients. Archives of Clinical Neuropsychology, 11(6), 513–519. doi:
10.1016/0887-6177(96)82330-4
Kanser, R. J., Rapport, L. J., Bashem, J. R., Billings, N. M., Hanks,
R. A., Axelrod, B. N., & Miller, J. B. (2017). Strategies of successful
and unsuccessful simulators coached to feign traumatic brain injury.
The Clinical Neuropsychologist, 31(3), 644–653. doi:10.1080/
13854046.2016.1278040
Kanser, R. J., Rapport, L. J., Bashem, J. R., & Hanks, R. A. (2019).
Detecting malingering in traumatic brain injury: Combining
response time with performance validity test accuracy. The Clinical
Neuropsychologist, 33(1), 90–107. doi:10.1080/13854046.2018.1440006
Kiewel, N. A., Wisdom, N. M., Bradshaw, M. R., Pastorek, N. J., &
Strutt, A. M. (2012). Retrospective review of Digit Span-related
effort indicators in probable Alzheimer’s disease patients. The
Clinical Neuropsychologist, 26(6), 965–974. doi:10.1080/13854046.
2012.694478
Kim, M. S., Boone, K. B., Victor, T., Marion, S. D., Amano, S.,
Cottingham, M. E., … Zeller, M. A. (2010). The Warrington
Recognition Memory Test for words as a measure of response bias:
Total score and response time cutoffs developed on “real world”
credible and noncredible subjects. Archives of Clinical
Neuropsychology, 25(1), 60–70. doi:10.1093/arclin/acp088
Kongs, S.K., Thompson, L.L., Iverson, G.L., & Heaton, R.K. (2000).
WCST-64: Wisconsin Card Sorting Test-64 card version professional
manual. Odessa, FL: Psychological Assessment Resources.
Kura, K. (2013). Japanese north–south gradient in IQ predicts differen-
ces in stature, skin color, income, and homicide rate. Intelligence,
41(5), 512–516. doi:10.1016/j.intell.2013.07.001
Lange, R. T., Iverson, G. L., Brickell, T. A., Staver, T., Pancholi, S.,
Bhagwat, A., & French, L. M. (2013). Clinical utility of the Conners’
Continuous Performance Test-II to detect poor effort in U.S. mili-
tary personnel following traumatic brain injury. Psychological
Assessment, 25 (2), 339–352. doi:10.1037/a0030915
Larochette, A. C., & Harrison, A. G. (2012). Word Memory Test per-
formance in Canadian adolescents with learning disabilities: A pre-
liminary study. Applied Neuropsychology: Child, 1(1), 38–47.
Larrabee, G. J. (2003). Detection of malingering using atypical perform-
ance patterns on standard neuropsychological tests. The Clinical
Neuropsychologist, 17(3), 410–425. doi:10.1076/clin.17.3.410.18089
16 J. HURTUBISE ET AL.
Larrabee, G. J. (2008). Aggregation across multiple indicators improves
the detection of malingering: Relationship to likelihood ratios. The
Clinical Neuropsychologist, 22(4), 666–679. doi:10.1080/
13854040701494987
Larrabee, G. J. (2014). False-positive rates associated with the use of
multiple performance and symptom validity tests. Archives of
Clinical Neuropsychology, 29(4), 364–373. doi:10.1093/arclin/acu019
Larrabee, G. J., Rohling, M. L., & Meyers, J. E. (2019). Use of multiple
performance and symptom validity measures: Determining the opti-
mal per test cutoff for determination of invalidity, analysis of skew,
and inter-test correlations in valid and invalid performance groups.
The Clinical Neuropsychologist, 33(8), 1354–1372. doi:10.1080/
13854046.2019.1614227
Larsen, J. D., Allen, M. D., Bigler, E. D., Goodrich-Hunsaker, N. J., &
Hopkins, R. O. (2010). Different patterns of cerebral activation in
genuine and malingered cognitive effort during performance on the
Word Memory Test. Brain Injury, 24(2), 89–99. doi:10.3109/
02699050903508218
Leighton, A., Weinborn, M., & Maybery, M. (2014). Bridging the gap
between neurocognitive processing theory and performance validity
assessment among the cognitively impaired: A review and methodo-
logical approach. Journal of the International Neuropsychological
Society, 20(9), 873–886. doi:10.1017/S135561771400085X
Le
on, F. R., & Leon, A. B. (2014). Why complex cognitive ability
increases with absolute latitude. Intelligence, 46, 291–299. doi:10.
1016/j.intell.2014.07.011
Lichtenstein, J. D., Erdodi, L. A., & Linnea, K. S. (2017). Introducing a
forced-choice recognition task to the California Verbal Learning
Test – Children’s Version. Child Neuropsychology, 23(3), 284–299.
doi:10.1080/09297049.2015.1135422
Lichtenstein, J. D., Flaro, L., Baldwin, F., Rai, J. K., & Erdodi, L. A.
(2019). Further evidence for embedded validity tests in children
within the Conners’ Continuous Performance Test – Second
Edition. Developmental Neuropsychology, 44(2), 159–171. doi:10.
1080/87565641.2019.1565536
Lichtenstein, J. D., Greenacre, M. K., Cutler, L., Abeare, K., Baker,
S. D., Kent, K. J., … Erdodi, L. A. (2019). Geographic variation and
instrumentation artifacts: In search of confounds in performance
validity assessment in adults with mild TBI. Psychological Injury and
Law, 12(2), 127–145. doi:10.1007/s12207-019-09354-w
Lichtenstein, J. D., Holcomb, M., & Erdodi, L. A. (2018). One-Minute
PVT: Further evidence for the utility of the California Verbal
Learning Test—Children’s Version Forced Choice Recognition Trial.
Journal of Pediatric Neuropsychology, 4(3-4), 94–104. doi:10.1007/
s40817-018-0057-4
Lichtenstein, J. D., Linnea, K. S., & Maerlender, A. C. (2018). Patterns
of referral in high school concussion management programs: A pilot
study of consultants from different disciplines. Applied
Neuropsychology: Child, 7(4), 334–341. doi:10.1080/21622965.2017.
1340158
Lippa, S. M. (2018). Performance validity testing in neuropsychology:
A clinical guide, critical review, and update on a rapidly evolving lit-
erature. The Clinical Neuropsychologist, 32(3), 391–421. doi:10.1080/
13854046.2017.1406146
Loring, D. W., Meador, K. J., Lee, G. P., King, D. W., Nichols, M. E.,
Park, Y. D., … Smith, J. R. (1995). Wada memory asymmetries pre-
dict verbal memory decline after anterior temporal lobectomy.
Neurology, 45(7), 1329–1333. doi:10.1212/WNL.45.7.1329
Lupu, T., Elbaum, T., Wagner, M., & Braw, Y. (2018). Enhanced detec-
tion of feigned cognitive impairment using per item response time
measurements in the Word Memory Test. Applied Neuropsychology:
Adult, 25(6), 532–542. doi:10.1080/23279095.2017.1341410
Lynn, R. (2010). In Italy, north–south differences in IQ predict differ-
ences in income, education, infant mortality, stature, and literacy.
Intelligence, 38(1), 93–100. doi:10.1016/j.intell.2009.07.004
MacAllister, W. S., Vasserman, M., & Armstrong, K. (2019). Are we
documenting performance validity testing in pediatric neuropsycho-
logical assessment? A brief report. Child Neuropsychology, 25(8),
1035–1042. doi:10.1080/09297049.2019.1569606
Martin, P. K., Hunter, B. P., Rach, A. M., Heinrichs, R. J., &
Schroeder, R. W. (2017). Excessive decline from premorbid func-
tioning: Detecting performance invalidity with the WAIS-IV and
demographic predictions. The Clinical Neuropsychologist, 31(5),
829–843. doi:10.1080/13854046.2017.1284265
Martin, P. K., Schroeder, R. W., & Odland, A. P. (2015).
Neuropsychologists’ validity testing beliefs and practices: A survey of
North American Professionals. The Clinical Neuropsychologist, 29(6),
741–746. doi:10.1080/13854046.2015.1087597
Martin, P. K., Schroeder, R. W., Wyman-Chick, K. A., Hunter, B. P.,
Heinrichs, R. J., & Baade, L. E. (2018). Rates of abnormally low
TOPF Word Reading scores in individuals failing versus passing
performance validity testing. Assessment, 25(5), 640–652. doi:10.
1177/1073191116656796
Mathias, J. L., Bowden, S. C., Bigler, E. D., & Rosenfeld, J. V. (2007). Is
performance on the Wechsler Test of Adult Reading affected by
traumatic brain injury?. British Journal of Clinical Psychology, 46(4),
457–466. doi:10.1348/014466507X190197
Matthews, C. G., & Klove, K. (1964). Instruction manual for the Adult
Neuropsychology Test Battery. Madison, WI: University of Wisconsin
Medical School.
McCrea, M., Guskiewicz, K. M., Marshall, S. W., Barr, W., Randolph,
C., Cantu, R. C., … Kelly, J. P. (2003). Acute effects and recovery
time following concussion in collegiate football players. JAMA,
290(19), 2556–2564. doi:10.1001/jama.290.19.2556
McDaniel, M. A. (2006). Estimating state IQ: Measurement challenges
and preliminary correlates. Intelligence, 34(6), 607–619. doi:10.1016/
j.intell.2006.08.007
McFarlane, J., Welch, J., & Rodgers, J. (2006). Severity of Alzheimer’s
disease and effect on premorbid measures of intelligence. British
Journal of Clinical Psychology, 45(4), 453–463. doi:10.1348/
014466505X71245
Merten, T., Bossink, L., & Schmand, B. (2007). On the limits of effort
testing: Symptom validity tests and severity of neurocognitive symp-
toms in nonlitigant patients. Journal of Clinical and Experimental
Neuropsychology, 29(3), 308–318. doi:10.1080/13803390600693607
Merten, T., & Rogers, R. (2017). An international perspective on
feigned mental disabilities: Conceptual issues and continuing contro-
versies. Behavioral Sciences & the Law, 35(2), 97–112. doi:10.1002/
bsl.2274
Meyers, J. E., Miller, R. M., Thompson, L. M., Scalese, A. M., Allred,
B. C., Rupp, Z. W., … Junghyun Lee, A. (2014). Using likelihood
ratios to detect invalid performance with performance validity meas-
ures. Archives of Clinical Neuropsychology, 29(3), 224–235. doi:10.
1093/arclin/acu001
Miller, L., Ryan, J., Carruthers, C., & Cluff, R. (2004). Brief screening
indexes for malingering: A confirmation of Vocabulary minus Digit
Span from the WAIS-III and the Rarely Missed Index from the
WMS-III. The Clinical Neuropsychologist, 18(2), 327–333. doi:10.
1080/13854040490501592
Nelson, N. W., Boone, K., Dueck, A., Wagener, L., Lu, P., & Grills, C.
(2003). The relationship between eight measures of suspect effort.
The Clinical Neuropsychologist, 17(2), 263–272. doi:10.1076/clin.17.2.
263.16511
Odland, A. P., Lammy, A. B., Martin, P. K., Grote, C. L., &
Mittenberg, W. (2015). Advanced administration and interpretation
of multiple validity tests. Psychological Injury and Law, 8(1), 46–63.
doi:10.1007/s12207-015-9216-4
Ord, J. S., Boettcher, A. C., Greve, K. J., & Bianchini, K. J. (2010).
Detection of malingering in mild traumatic brain injury with the
Conners’ Continuous Performance Test-II. Journal of Clinical and
Experimental Neuropsychology, 32(4), 380–387. doi:10.1080/
13803390903066881
Pearson. (2009). Advanced clinical solutions for the WAIS-IV and
WMS-IV – Technical manual. San Antonio, TX: Author.
Persinger, V. C., Whiteside, D. M., Bobova, L., Saigal, S. D., Vannucci,
M. J., & Basso, M. R. (2018). Using the California Verbal Learning
Test, Second Edition as an embedded performance validity measure
among individuals with TBI and individuals with psychiatric

disorders. The Clinical Neuropsychologist, 32(6), 1039–1053. doi:10.
1080/13854046.2017.1419507
Piatt, A. L., Fields, J. A., Paolo, A. M., Koller, W. C., & Troster, A. I.
(1999). Lexical, semantic, and action verbal fluency in Parkinson’s
disease with and without dementia. Journal of Clinical and
Experimental Neuropsychology, 21(4), 435–443. doi:10.1076/jcen.21.4.
435.885
Piatt, A. L., Fields, J. A., Paolo, A. M., & Troster, A. I. (1999). Action
(verb naming) fluency as an executive function measure: Convergent
and divergent evidence of validity. Neuropsychologia, 37(13),
1499–1503. doi:10.1016/S0028-3932(99)00066-4
Powell, M. R., Locke, D. E., Smigielski, J. S., & McCrea, M. (2011).
Estimating the diagnostic value of the trail making test for subopti-
mal effort in acquired brain injury rehabilitation patients. The
Clinical Neuropsychologist, 25(1), 108–118. doi:10.1080/13854046.
2010.532912
Poynter, K., Boone, K. B., Ermshar, A., Miora, D., Cottingham, M.,
Victor, T. L., … Wright, M. (2019). Wait, there’s a baby in this
bath water! Update on quantitative and qualitative cut-offs for Rey
15-Item Recall and Recognition. Archives of Clinical
Neuropsychology, 34(8), 1367–1380. doi:10.1093/arclin/acy087
Proto, D. A., Pastorek, N. J., Miller, B. I., Romesser, J. M., Sim, A. H.,
& Linck, J. M. (2014). The dangers of failing one or more perform-
ance validity tests in individuals claiming mild traumatic brain
injury-related postconcussive symptoms. Archives of Clinical
Neuropsychology, 29(7), 614–624. doi:10.1093/arclin/acu044
Rai, J., An, K. Y., Charles, J., Ali, S., & Erdodi, L. A. (2019).
Introducing a forced choice recognition trial to the Rey Complex
Figure Test. Psychology and Neuroscience. doi:10.1037/pne0000175
Rai, J., & Erdodi, L. (2019). The impact of criterion measures on the
classification accuracy of TOMM-1. Applied Neuropsychology: Adult,
doi:10.1080/23279095.2019.161.1613994
Rapport, L. J., Farchione, T. J., Coleman, R. D., & Axelrod, B. N.
(1998). Effects of coaching on malingered motor function profiles.
Journal of Clinical and Experimental Neuropsychology, 20(1), 89–97.
doi:10.1076/jcen.20.4.89.1143
Rey, A. (1941). L’examen psychologique dans les cas d’encephalopathie
traumatique [Psychological examination in cases of traumatic
encephalopathy]. Archives de Psychologie, 28, 286–340.
Rogers, R., Sewell, K. W., Martin, M. A., & Vitacco, M. J. (2003).
Detection of feigned mental disorders: A meta-analysis of the
MMPI-2 and malingering. Assessment, 10(2), 160–177. doi:10.1177/
1073191103010002007
Rosenfeld, B., Sands, S. A., & van Gorp, W. G. (2000). Have we forgot-
ten the base rate problem? Methodological issues in the detection of
distortion. Archives of Clinical Neuropsychology, 15(4), 349–359. doi:
10.1016/S0887-6177(99)00025-6
Roye, S., Calamia, M., Bernstein, J. P., De Vito, A. N., & Hill, B. D.
(2019). A multi-study examination of performance validity in under-
graduate research participants. The Clinical Neuropsychologist, 33(6),
1138–1155. doi:10.1080/13854046.2018.1520303
Rose, F. E., Hall, S., Szalda-Petree, A. D., & Bach, P. (1998). A com-
parison of four tests of malingering and the effects of coaching.
Archives of Clinical Neuropsychology, 13(4), 349–363. doi:10.1016/
S0887-6177(97)00025-5
Roth, R. M., Erdodi, L. A., McCulloch, L. J., & Isquith, P. K. (2015).
Much ado about norming the Behavior Rating Inventory of
Executive Function. Child Neuropsychology, 21(2), 225–233. doi:10.
1080/09297049.2014.897318
Savla, G., Twamley, E. W., Thompson, W. K., Delis, D. C., Jeste, D. V.,
& Palmer, B. W. (2011). Evaluation of specific executive functioning
skills and the processes underlying executive control in schizophre-
nia. Journal of the International Neuropsychological Society, 17(01),
14–23. doi:10.1017/S1355617710001177
Sawyer, R. J., Testa, S. M., & Dux, M. (2017). Embedded performance
validity tests within the Hopkins Verbal Learning Test – Revised
and the Brief Visuospatial Memory Test – Revised. The Clinical
Neuropsychologist, 31(1), 207–218. doi:10.1080/13854046.2016.
1245787
APPLIED NEUROPSYCHOLOGY: CHILD 17
Schroeder, R. W., & Marshall, P. S. (2010). Validation of the Sentence
Repetition Test as a measure of suspect effort. The Clinical
Neuropsychologist, 24(2), 326–343. doi:10.1080/13854040903369441
Schroeder, R. W., Martin, P. K., Heinrichs, R. J., & Baade, L. E. (2019).
Research methods in performance validity testing studies: Criterion
grouping approach impacts study outcomes. The Clinical
Neuropsychologist, 33(3), 466–477. doi:10.1080/13854046.2018.
1484517
Schroeder, R. W., Twumasi-Ankrah, P., Baade, L. E., & Marshall, P. S.
(2012). Reliable Digit Span: A systematic review and cross-validation
study. Assessment, 19(1), 21–30. doi:10.1177/1073191111428764
Schutte, C., Axelrod, B. N., & Montoya, E. (2015). Making sure neuro-
psychological data are meaningful: Use of performance validity test-
ing in medicolegal and clinical contexts. Psychological Injury and
Law, 8(2), 100–105. doi:10.1007/s12207-015-9225-3
Schwartz, E. S., Erdodi, L., Rodriguez, N., Jyotsna, J. G., Curtain, J. R.,
Flashman, L. A., & Roth, R. M. (2016). CVLT-II forced choice rec-
ognition trial as an embedded validity indicator: A systematic review
of the evidence. Journal of the International Neuropsychological
Society, 22(8), 851–858. doi:10.1017/S1355617716000746
Shura, R. D., Miskey, H. M., Rowland, J. A., Yoash-Gantz, R. E., &
Denning, J. H. (2016). Embedded performance validity measures
with postdeployment veterans: Cross-validation and efficiency with
multiple measures. Applied Neuropsychology: Adult, 23(2), 94–104.
doi:10.1080/23279095.2015.1014556
Shura, R. D., Martindale, S. L., Taber, K. H., Higgins, A. M., &
Rowland, J. A. (2019). Digit Span embedded validity indicators in
neurologically-intact veterans. The Clinical Neuropsychologist. doi:10.
1080/13854046.2019.1635209
Silverberg, N. D., Hanks, R. A., Buchanan, L., Fichtenberg, N., &
Millis, S. R. (2008). Detecting response bias on an expanded version
of the controlled word association test. The Clinical
Neuropsychologist, 22(1), 140–157. doi:10.1080/13854040601160597
Slick, D. J., Sherman, E. M. S., & Iverson, G. L. (1999). Diagnostic cri-
teria for malingered neurocognitive dysfunction: Proposed standards
for clinical practice and research. The Clinical Neuropsychologist,
13(4), 545–561. doi:10.1076/1385-4046(199911)13:04;1-Y;FT545
Smith, A. (2007). Symbol Digit Modalities Test. Torrance, CA: Western
Psychological Services.
Spencer, R. J., Axelrod, B. N., Drag, L. L., Waldron-Perrine, B.,
Pangilinan, P. H., & Bieliauskas, L. A. (2013). WAIS-IV Reliable
Digit Span is no more accurate than Age Corrected Scaled Score as
an indicator of invalid performance in a veteran sample undergoing
evaluation for mTBI. The Clinical Neuropsychologist, 27(8),
1362–1372. doi:10.1080/13854046.2013.845248
Stevens, A., Friedel, E., Mehren, G., & Merten, T. (2008). Malingering
and uncooperativeness in psychiatric and psychological assessment:
Prevalence and effects in a German sample of claimants. Psychiatry
Research, 157(1–3), 191–200. doi:10.1016/j.psychres.2007.01.003
Steward, K. A., Novack, T. A., Kennedy, R., Crowe, M., Marson, D. C.,
& Triebel, K. L. (2017). The Wechsler Test of Adult Reading as a
measure of premorbid intelligence following traumatic brain injury.
Archives of Clinical Neuropsychology, 32(1), 98–103. doi:10.1093/
arclin/acw081
Suchotzki, K., Crombez, G., Smulders, F. T., Meijer, E., & Verschuere,
B. (2015). The cognitive mechanisms underlying deception: An
event-related potential study. International Journal of
Psychophysiology, 95(3), 395–405. doi:10.1016/j.ijpsycho.2015.01.010
Sugarman, M. A., & Axelrod, B. N. (2015). Embedded measures of per-
formance validity using verbal fluency tests in a clinical sample.
Applied Neuropsychology: Adult, 22(2), 141–146. doi:10.1080/
23279095.2013.873439
Suhr, J. A., & Boyer, D. (1999). Use of the Wisconsin Card Sorting
Test in the detection of malingering in student simulator and
patient samples. Journal of Clinical and Experimental
Neuropsychology, 21(5), 701–708. doi:10.1076/jcen.21.5.701.868
Tomer, E., Lupu, T., Golan, L., Wagner, M., & Braw, Y. (2019). Eye
tracking as a mean to detect feigned cognitive impairment in the
Word Memory Test. Applied Neuropsychology: Adult, 27(1), 49–61.
doi:10.1080/23279095.2018.1480483
18 J. HURTUBISE ET AL.
Trueblood, W. (1994). Qualitative and quantitative characteristics of
malingered and other invalid WAIS-R and clinical memory data.
Journal of Clinical and Experimental Neuropsychology, 16(4),
597–607. doi:10.1080/088639408402671
Tsushima, W. T., Yamamoto, M. H., Ahn, H. J., Siu, A. M., Choi,
S. Y., & Murata, N. M. (2019). Invalid baseline testing with
ImPACT: Does sandbagging occur with high school athletes?
Applied Neuropsychology: Child, 1–10.
Tyson, B. T., Baker, S., Greenacre, M., Kent, K. J., Lichtenstein, J. D.,
Sabelli, A., & Erdodi, L.A. (2018). Differentiating epilepsy from psy-
chogenic nonepileptic seizures using neuropsychological test data.
Epilepsy & Behavior, 87, 39–45. doi:10.1016/j.yebeh.2018.08.010
Vickery, C. D., Berry, D. T. R., Dearth, C. S., Vagnini, V. L., Baser,
R. E., Cragar, D. E., & Orey, S. A. (2004). Head injury and the abil-
ity to feign neuropsychological deficits. Archives of Clinical
Neuropsychology, 19(1), 37–48. doi:10.1093/arclin/19.1.37
Viglione, D. J., Giromini, L., Landis, P., McCullaugh, J. M., Pizitz,
T. D., O’Brien, S., … Abramsky, A. (2019). Development and valid-
ation of the False Disorder Score: The Focal Scale of the Inventory
of Problems–29. Journal of Personality Assessment, 101(6), 653–661.
doi:10.1080/00223891.2018.1492413
von Helvoort, D., Merckelbach, H., & Merten, T. (2019). The Self-
Report Symptom Inventory (SRSI) is sensitive to instructed feigning,
but not to genuine psychopathology in male forensic inpatients: An
initial study. The Clinical Neuropsychologist, 33(6), 1069–1082. doi:
10.1080/13854046.2018.1559359
Webber, T. A., & Soble, J. R. (2018). Utility of various WAIS-IV Digit
Span indices for identifying noncredible performance among per-
formance validity among cognitively impaired and unimpaired
examinees. The Clinical Neuropsychologist, 32 (4), 657–670. doi:10.
1080/13854046.2017.1415374
Wechsler, D. (1997). Wechsler Adult Intelligence Scale (3rd ed.). San
Antonio, TX: The Psychological Corporation.
Whiteside, D. M., Kogan, J., Wardin, L., Phillips, D., Franzwa, M. G.,
Rice, L., … Roper, B. (2015). Language-based embedded perform-
ance validity measures in traumatic brain injury. Journal of Clinical
and Experimental Neuropsychology, 37(2), 220–227. doi:10.1080/
13803395.2014.1002758
Whitney, K. A., Davis, J. J., Shepard, P. H., Bertram, D. M., & Adams,
K. M. (2009). Digit span age scaled score in middle-aged military
veterans: Is it more closely associated with TOMM failure than reli-
able digit span?. Archives of Clinical Neuropsychology, 24(3),
263–272. doi:10.1093/arclin/acp034
Whiteside, D. M., Caraher, K., Hahn-Ketter, A., Gaasedelen, O., &
Basso, M. R. (2019). Classification accuracy of individual and com-
bined executive functioning embedded performance validity meas-
ures in mild traumatic brain injury. Applied Neuropsychology: Adult,
26(5), 472–481. doi:10.1080/23279095.2018.1443935
Whiteside, D. M., Gaasedelen, O. J., Hahn-Ketter, A. E., Luu, H.,
Miller, M. L., Persinger, V., … Basso, M. R. (2015). Derivation of a
cross-domain embedded performance validity measure in traumatic
brain injury. The Clinical Neuropsychologist, 29(6), 788–803. doi:10.
1080/13854046
Wilkinson, G. S., & Robertson, G. J. (2006). Wide range achievement
test 4. Lutz, FL: Psychological Assessment Resources, Inc.
Yu, J., Tao, Q., Zhang, R., Chan, C. C. H., & Lee, T. M. C. (2019). Can
fMRI discriminate between deception and false memory? A meta-
analytic comparison between deception and false memory studies.
Neuroscience & Biobehavioral Reviews, 104, 43–55. doi:10.1016/j.neu-
biorev.2019.06.027
Zakzanis, K. K., McDonald, K., & Troyer, A. K. (2011). Component
analysis of verbal fluency in patients with mild traumatic brain
injury. Journal of Clinical and Experimental Neuropsychology, 33(7),
785–792. doi:10.1080/13803395.2011.558496
Zakzanis, K. K., McDonald, K., & Troyer, A. K. (2013). Component
analysis of verbal fluency in patients with mild traumatic brain
injury. Brain Injury, 27(7–8), 903–908. doi:10.3109/02699052.2013.
775505
Zuccato, B. G., Tyson, B. T., & Erdodi, L. A. (2018). Early bird fails
the PVT? The effects of timing artifacts on performance validity
tests. Psychological Assessment, 30(11), 1491–1498. doi:10.1037/
pas0000596