NIH Toolbox for Assessment of Neurological and Behavioral Function--Cognition Battery, V3

Mild & Moderate/Severe Stroke: (Carlozzi et al, 2017a; n = 131 (n = 71 mild stroke; n = 60 moderate/severe stroke—54 moderate and 6 severe); mean age = 57.5 (12.6) years; age range = 22-83 years; male = 51%; median time post CVA = 29.0 months (range = 12.5-87.3 months); mean = 31.5 (11.8) months)

National Institute of Health (NIH) Toolbox (NIHTB) – Cognition Battery T Scores for Individuals with Mild vs Moderate/Severe Stroke 

*Fluid cognitive composite score combines Dimensional Change Card Sort (DCCS) Test, Flanker Test of Executive Function Inhibitory Control and Attention, Picture Sequence Memory Test of episodic memory, List Sorting Working Memory Test, and Pattern Comparison Processing Speed Test. Crystallized cognitive composite score includes Picture Vocabulary and Oral Reading scores.

Stroke: (Carlozzi et al, 2017b; n = 211; mean age = 56.13 (12.97); female = 50.2%, mean time post CVA = 2.74 (2.46) years)

NIHTB Cognition Battery T Scores for Stroke Participants^a

^aT scores are demographically adjusted for age, sex, education, and race/ethnicity.

^b% impairment reflects individuals with scores >1 SD beyond the mean on a given test in the negative direction.

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Stroke: (Carlozzi et al, 2017a)

Effect sizes (Cohen’s d) by stroke severity for NIHTB – Cognition Battery composite and subtest scores

Brain Injury: (Tulskey et al, 2017; n = 182 (mild or moderate, n = 83 or severe, n = 99); mean age = 38.6 (17.4) years; mean time since TBI = 5.8 years (5.6) years).

• 16^th percentile (T score ≤ 40) is the cutoff for a low score. Complicated mild/moderate TBI and severe TBI respectively had individuals with 59% and 75% of individuals have one or more low scores.  Normative base rate of 46% suggests that .54 specificity is poor. 

Brain Injury: (Tulskey et al, 2017)

National Institute of Health (NIH) Toolbox (NIHTB) Demographically Adjusted T  Scores for Individuals with Complicated Mild/Moderate TBI, Severe TBI, and Matched Controls 

*Fluid cognitive composite score combines Dimensional Change Card Sort (DCCS) Test, Flanker Test of Executive Function Inhibitory Control and Attention, Picture Sequence Memory Test of episodic memory, List Sorting Working Memory Test, and Pattern Comparison Processing Speed Test. Crystallized cognitive composite score includes Picture Vocabulary and Oral Reading scores.

Brain Injury: (Carlozzi et al, 2017); n = 184; mean age = 39.83 (17.4) years; male = 64.1%; mean time since TBI = 5.95 (5.54) years)

NIHTB Cognition T Scores for TBI Participants^a

^aT scores are demographically adjusted for age, sex, education, and race/ethnicity.

^b% impairment reflects individuals with scores >1 SD beyond the mean on a given test in the negative direction.

Convergent validity: 

Brain Injury: (Tulskey et al., 2017)

• Excellent correlation between Oral Reading Recognition and Wide Range Achievement Test, 4^th Edition (r = 0.83)
• Excellent correlation between Picture Vocabulary and Peabody Picture Vocabulary Test, 4^th Edition (r = 0.80)
• Adequate correlation between List Sorting and WAIS-IV Letter Number Sequencing (r = 0.56)
• Excellent correlation between Picture Sequence Memory and Brief Visuospatial Memory Test-Revised (r = 0.68)
• Excellent correlation between Pattern Comparison and WAIS-IV Coding (r = 0.69)
• Adequate correlation between Flanker Inhibitory Control and DKEFS Color-Word Interference-Inhibition (r = -0.46)
• Adequate correlation between Dimensional Change Card Sort and Wisconsin Card Sort Test (r = -0.42)

Discriminant validity:

Repetitive Head Impact (RHI): (Amadon et al., 2023; n = 176, mean age = 21.19 (1.63), female = 60 (34%) (n = 115 in contact sport, mean age = 21.46 (1.65), female = 20 (17%); and 61 in non-contact sport, mean age = 20.67 (1.45), female = 40 (66%))

• Significant discriminate ability across the whole sample of the Picture Sequence Memory Test (episodic memory) to distinguish between contact sport and non-contact sport athletes (F(1,171) = 7.16, p = 0.008), years of exposure (F(1,171) = 4.19, p = 0.042), and extensive vs. non-extensive RHI exposure based on traumatic encephalopathy syndrome (TES) criteria (F(1,171) = 5.12, p = 0.025) 
  • A similar—although not significant--effect was observed for age of first exposure (AFE), with athletes with AFE < 12 having better episodic memory than those with AFE 12+/none (F(1,171) = 3.88, p = 0.05)
• Significant discriminate ability among females of the Picture Sequence Memory Test (episodic memory) to distinguish between contact sport and non-contact sport athletes (F(1,56) = 5.05, p = 0.03), years of exposure (F(1,56) = 4.54, p = 0.04), athletes with AFE < 12 vs. those with AFE 12+/none (F(1,56) = 4.76, p = 0.03), and extensive vs. non-extensive RHI exposure based on TES criteria (F(1,56) = 4.99, p = 0.03) 
• Although the direction of the relationships was the same as those seen in women, among males there were no significant relationships between RHI measures and episodic memory performance. 

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Intellectual Disability: (Shields et al., 2020; n = 242, mean age = 15.71 (5.15) years (Down syndrome = 91, mean age = 15.88 (5.17) years; Fragile X syndrome = 75, mean age = 16.16 (4.92) years; Other ID = 76, mean age = 15.05 (5.35) years); chronological age from 6 through 25 years; full-scale IQ <80 on Stanford-Binet, 5^th edition (SB-5); mental age of at least 3.0 years on the SB-5)  

• Acceptable Total test-retest reliability for Flanker Inhibitory Control and Attention (ICC = 0.74)
• Acceptable test-retest reliability for Dimensional Change Card Sort (ICC = 0.71)
• Acceptable Total test-retest reliability for List Sorting Working Memory (ICC = 0.74)
• Acceptable Total test-retest reliability for Pattern Comparison (ICC = 0.77)
• Poor Total test-retest reliability for Picture Sequence Memory (ICC = 0.47 with form A-A and ICC = 0.55 with form A-B) 
• Acceptable Total test-retest reliability for Picture Vocabulary (ICC = 0.85)
• Excellent Total test-retest reliability for Oral Reading and Recognition (ICC = 0.96)
• Acceptable Total test-retest reliability for Fluid Composite (ICC = 0.83)
• Excellent Total test-retest reliability for Crystalized Composite (ICC = 0.93)
• Excellent Total test-retest reliability for Cognitive Function Composite (ICC = 0.92)

Intellectual Disability: (Hessl et al., 2016; Fragile X Syndrome: n = 63, mean age = 19.3 (8.3) years, mean mental age = 5.3 (1.6) years )

• Acceptable test-retest validity for Flanker Inhibitory Control and Attention (ICC = 0.75)
• Acceptable test-retest validity for Dimensional Change Card Sort (ICC = 0.88)
• Acceptable test-retest validity for List Sorting Working Memory (ICC = 0.84)
• Excellent test-retest validity for Pattern Comparison (ICC = 0.90)
• Acceptable test-retest validity for Picture Sequence Memory (ICC = 0.76) 
• Acceptable test-retest validity for Picture Vocabulary (ICC = 0.77)
• Excellent test-retest validity Oral Reading and Recognition (ICC = 0.99)

Convergent validity:

Intellectual Disability: (Shields et al., 2020)

• Adequate convergent validity of Flanker Inhibitory Control and Attention with Conners Kiddie Continuous Performance Test, 2^nd Edition (r = -0.52*)
• Adequate convergent validity of Dimensional Change Card Sort with NEPSY Inhibition (NEPSY-IN) subtest (r = 0.48*)
• Excellent convergent validity of List Sorting Working Memory with Stanford-Binet, 5^th Edition (SB-5) Verbal Working Memory (r = 0.65*) 
• Excellent convergent validity of Pattern Comparison with Wechsler Preschool and Primary Scale of Intelligence, 4^th Edition Bug Search (r = 0.66*) 
• Adequate convergent validity of Picture Sequence Memory with Leiter International Performance Scale, 3^rd Edition Forward Memory (Leiter-FM)(r = 0.47*) 
• Excellent convergent validity of Picture Vocabulary Test with Peabody Picture Vocabulary Test, 4^th Edition (r = 0.83*)
• Excellent convergent validity of Oral Reading and Recognition with Woodcock Johnson 4^th Edition Letter-Word Identification (WJ-LW) (r = 0.92*)
• Excellent convergent validity of Fluid Composite with SB-5 Fluid Reasoning IQ (r = 0.60*)
• Excellent convergent validity of Crystalized Composite with SB-5 Verbal IQ (r = 0.75*)

*Significant at p < 0.001

Discriminant validity:

Intellectual Disability: (Shields, et al, 2020)

• Adequate discriminant validity of Flanker Inhibitory Control and Attention with WJ-LW (r = -0.53*)
• Adequate discriminant validity of Dimensional Change Card Sort with WJ-LW (r = 0.36*)
• Adequate discriminant validity of List Sorting Working Memory with WJ-LW (r = 0.49*)
• Adequate discriminant validity of Pattern Comparison with WJ-LW (r = 0.45*) 
• Adequate discriminant validity of Picture Sequence Memory with WJ-LW (r = 0.50*) 
• Adequate discriminant validity of Picture Vocabulary Test with Leiter-FM (r = 0.47*)
• Adequate discriminant validity of Oral Reading and Recognition with Leiter-FM (r = 0.58*)
• Poor discriminant validity of Fluid Composite with SB-5 Verbal IQ (r = 0.61*)
• Poor discriminant validity of Crystalized Composite with SB-5 Fluid Reasoning IQ (r = 0.68*)

*Significant at p < 0.001

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Mixed populations: (Weintraub et al, 2013b; n = 476, age range = 3-85 years, female = 53.1%, stratified sample of community-dwelling individuals)

• Excellent  test-retest validity for Flanker Inhibitory Control and Attention (ICC = 0.9­6)
• Excellent test-retest validity for Dimensional Change Card Sort (ICC = 0.94)
• Acceptable test-retest validity for List Sorting Working Memory (ICC = 0.89)
• Acceptable  test-retest validity for Pattern Comparison (ICC = 0.82)
• Acceptable test-retest validity for Picture Sequence Memory (ICC = 0.78)
• Excellent  test-retest validity for Picture Vocabulary (ICC = 0.94)
• Excellent test-retest validity for Oral Reading and Recognition (ICC = 0.99)

Convergent validity:

Mixed populations: (Weintraub et al., 2013b, age range = 8-85)

• Adequate  convergent validity of Flanker Inhibitory Control and Attention with WISC-IV/WAIS-IV Letter-Number Sequencing, Coding, Symbol Search average^a (r = -0.48^b)
• Adequate  convergent validity of Dimensional Change Card Sort with D-KEFS Inhibition (r = -0.51^b)
• Adequate  convergent validity of List Sorting Working Memory with WISC-IV/WAIS-IV Letter-Number Sequencing^a/Paced Auditory Serial Addition Test average (r = 0.58^b)
• Adequate,  convergent validity of Pattern Comparison with WISC-IV/WAIS-IV Coding/Symbol Search average^a (r = 0.49^b)
• Excellent  convergent validity of Picture Sequence Memory with  BVMT-R/Rey Auditory Verbal Learning Test average^a (RALVT) 0.69^b)
• Excellent  convergent validity of Picture Vocabulary with PPVT-4 (r = 0.78^b)
• Excellent  convergent validity of Oral Reading and Recognition with Wide Range Achievement Test, 4^th Editon (WRAT-4) (r = 0.93^b)

Discriminant validity:

Mixed populations: (Weintraub et al., 2013b, age range = 8-85) 

• Excellent  discriminant validity of Flanker Inhibitory Control and Attention with PPVT-4 (r = 0.15^c)
• Excellent  discriminant validity of Dimensional Change Card Sort with PPVT-4 (r = 0.14^d)
• Excellent  discriminant validity of List Sorting Working Memory with PPVT-4 (r = 0.30^b)
• Excellent  discriminant validity of Pattern Comparison with PPVT-4 (r = 0.12^d)
• Excellent  discriminant validity of Picture Sequence Memory with PPVT-4 (r = -0.08)
• Excellent  discriminant validity of Picture Vocabulary with BVMT-R/RAVLT average^a (r = 0.08)
• Excellent  discriminant validity of Oral Reading and Recognition with BVMT-R/RAVLT average^a (r = 0.19^b)

^aMeasure used dependent upon subjects’ age.

^bp < 0.001

^cp < 0.01

^dp < 0.05

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Multiple Sclerosis: (Manglani et al., 2022; n = 87; age range = 30-59; subjects diagnosed with relapsing-remitting MS; participating in a randomized control trial comparing effects of a physical activity intervention with a water consumption intervention on cognition)

Unadjusted standard scores on NIH Toolbox Cognition Battery

^aDimensional Change Card Sort

^bDiscriminant validity measure

Convergent validity:

Multiple Sclerosis: (Manglani et al., 2022)

• Adequate convergent validity of the pattern comparison (Processing Speed) subtest with the WAIS-IV Cancelation Test (mean concordance correlation coefficient (Mccc) = 0.46)
• Adequate convergent validity of the list sorting (Working Memory) subtest with the WAIS-IV Digit Span Test (Mccc = 0.34)
• Adequate convergent validity of the picture sequence (Episodic Memory) subtest with the California Verbal Learning Test, Second Edition (Mccc = 0.33)
• Poor convergent validity of the DCCS and the Flanker Test (Mccc = 0.20) (both Executive Function) with the Delis-Kaplan Executive Function System Sorting Test (DKEFS)

Discriminant validity:

Multiple Sclerosis: (Manglani et al., 2022)

• Excellent discriminant validity of the pattern comparison subtest (Processing Speed) with the Picture Vocabulary Test (concordance correlation coefficient (CCC) = 0.071)
• Excellent discriminant validity of the list sorting (Working Memory) pattern comparison subtest with the Picture Vocabulary Test (CCC = 0.29)
• Excellent discriminant validity of the picture sequence memory subtest with the Picture Vocabulary Test (CCC = 0.19)
• Excellent discriminant validity of the DCCS (CCC = 0.16) and the Flanker Test (CCC = 0.11) (both Executive Function) with the Picture Vocabulary Test

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Pediatric Epilepsy: (Matuska et al., 2024; n = 47; male = 55%; mean age = 10.0 (3.7) years; age range = 4.2-18.3 years; mean age of seizure onset = 4.9 (3.2) years; number of antiseizure medications at testing: 0 = 8.5%, 1 = 36.2%, 2 = 31.9%, 3 = 14.9%, 4 = 8.5%; history of Electrical Status Epilepticus in Sleep (ESES) = 46.8%)

Average performance of pediatric epilepsy patients on two NIH Toolbox Cognition Battery tests (n = 46)

Concurrent validity:

Pediatric Epilepsy: (Matuska et al., 2024)

• Adequate concurrent validity between Flanker Inhibitory Test and Pattern Comparison Test (r = 0.514, p < 0.001)
• Adequate concurrent validity between Pattern Comparison Test and Test of Everyday Attention for Children (TEA-Ch) (Sky Search – Accuracy) (r = 0.459, p < 0.05)
• Adequate concurrent validity between Flanker Inhibitory Test and Wechsler Intelligence Scale for Children (WISC) (Working Memory Index (WMI) (r = 0.455, p = 0.009)
• Adequate concurrent validity between Pattern Comparison Test and WISC (WMI) (r = 0.381, p = 0.035)
• Adequate concurrent validity between Flanker Inhibitory Test and Semantic Fluency (age-appropriate version of A Developmental NEuroPSYchological Assessment, Second Edition (NEPSY-II) or the Delis-Kaplan Executive Function System (D-KEFS)) (r = 0.373, p = 0.023)
• Adequate concurrent validity between Pattern Comparison Test and Semantic Fluency (age-appropriate version of NEPSY-II or D-KEFS (r = 0.365, p = 0.028)
• Adequate concurrent validity between Flanker Inhibitory Test and Comprehensive Test of Phonological Processing, Second Edition (CTOPP-2) (r = 0.577, p = 0.001)
• Adequate concurrent validity between Flanker Inhibitory Test and Wechsler Intelligence Scale for Children (WISC) (Processing Speed Index (PSI) (r = 0.378, p = 0.030)
• Adequate concurrent validity between Flanker Inhibitory Test and Grooved Pegboard w/dominant hand (r = 0.554, p < 0.001) and non-dominant hand (r = 0.420, p = 0.008)
• Adequate concurrent validity between Flanker Inhibitory Test and Verbal IQ (r = 0.458, p = 0.002)
• Adequate concurrent validity between Flanker Inhibitory Test and Nonverbal IQ (r = 0.495, p = 0.002)
• Adequate concurrent validity between Pattern Comparison Test and Nonverbal IQ (r = 0.483, p = 0.002)
• Agreement statistics conducted between the two NIHTB-CB tasks and the above clinical measures found that overall agreement ranged from 57 to 74% for Flanker and from 52 to 72% for Pattern Comparison.
  • Sensitivity was lower for Flanker (42-67%) than for Pattern Comparison (45-83%)
  • Specificity was higher for Flanker (70-83%) than for Pattern Comparison (48-83%)

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a).