NIH Toolbox for Assessment of Neurological and Behavioral Function--Cognition Battery, V3

Mild & Moderate/Severe Stroke: (Carlozzi et al, 2017a; n = 131 (n = 71 mild stroke; n = 60 moderate/severe stroke—54 moderate and 6 severe); mean age = 57.5 (12.6) years; age range = 22-83 years; male = 51%; median time post CVA = 29.0 months (range = 12.5-87.3 months); mean = 31.5 (11.8) months)

National Institute of Health (NIH) Toolbox (NIHTB) – Cognition Battery T Scores for Individuals with Mild vs Moderate/Severe Stroke 

NIHTB scores

n

Mild stroke

Mean (SD)

n

Moderate/severe stroke

Mean (SD)

Composite scores*

   Fluid

71

42.71 (12.64)

42

34.00 (9.57)

   Crystallized

75

50.54 (11.73)

50

45.72 (10.85)

Subtest scores

   Picture vocabulary

77

50.65 (12.77)

51

47.08 (11.38)

   Oral reading recognition

75

50.10 (10.18)

50

45.22 (10.76)

   Picture sequence memory test

73

45.86 (12.96)

45

35.98 (11.34)

   Pattern comparison

74

45.10 (11.25)

49

38.05 (9.13)

   List sorting

74

45.70 (10.26)

45

42.21 (10.86)

   Flanker

77

44.74 (10.89)

50

37.76 (9.53)

   DCCS

77

44.31 (10.11)

49

38.94 (8.83)

*Fluid cognitive composite score combines Dimensional Change Card Sort (DCCS) Test, Flanker Test of Executive Function Inhibitory Control and Attention, Picture Sequence Memory Test of episodic memory, List Sorting Working Memory Test, and Pattern Comparison Processing Speed Test. Crystallized cognitive composite score includes Picture Vocabulary and Oral Reading scores.

Stroke: (Carlozzi et al, 2017b; n = 211; mean age = 56.13 (12.97); female = 50.2%, mean time post CVA = 2.74 (2.46) years)

NIHTB Cognition Battery T Scores for Stroke Participantsa

NIHTB scores

n

Mean (SD)

% Impairedb

Composite scores

   Fluid

176

40.51 (11.59)

49.2

   Crystallized

176

49.54 (10.85)

19.2

Subtest scores

   Picture vocabulary

201

49.08 (12.06)

22.3

   Oral reading recognition

201

48.34 (10.58)

23.3

   Picture sequence memory test

175

42.64 (12.38)

43.8

   Pattern comparison

175

43.13 (10.74)

39.2

   List sorting

175

43.98 (9.96)

40.3

   Flanker

175

42.08 (10.58)

40.3

   DCCS

175

42.46 (9.47)

40.3

aT scores are demographically adjusted for age, sex, education, and race/ethnicity.

b% impairment reflects individuals with scores >1 SD beyond the mean on a given test in the negative direction.

Convergent validity:

Stroke: (Carlozzi et al, 2017a; all correlations significant at p < 0.01)

• Adequate convergent validity between Flanker and Delis Kaplan Executive Functioning System (DKEFS) Interference (r = 0.46; r = 0.50 with motor function included as a covariate)
• Adequate convergent validity between Dimensional Change Card Sort (DCCS) and DKEFS Interference (r = 0.54; r = 0.40 with motor function included as a covariate)
• Excellent convergent validity between List Sorting Working Memory and Wechsler Adult Intelligence Scale Letter Number Sequencing, Fourth Edition (WAIS-IV LN) (r = 0.66)
• Adequate convergent validity between Pattern Comparison and WAIS-IV Coding (CD) (r = 0.59; r = 0.58 with motor function included as a covariate)
• Excellent convergent validity between Pattern Comparison and WAIS-IV Symbol Search (SS) (r = 0.67; r = 0.66 with motor function included as a covariate)
• Adequate convergent validity between Picture Sequence Memory and Auditory Verbal Learning Test (Rey) (AVLT) (r = 0.52)
• Excellent convergent validity between Picture Sequence Memory and Brief Visuospatial Memory Test-Revised (BVMT-R) (r = 0.65)
• Excellent convergent validity between Picture Vocabulary and Peabody Picture Vocabulary Test, Fourth Edition (PPVT-IV) (r = 0.87)
• Excellent convergent validity between Oral Reading Recognition and Word Reading Achievement Test-Fourth Edition (WRAT-IV) (r = 0.88)

Discriminant validity

Stroke: (Carlozzi et al, 2017a) 

Convergent and Discriminant Validity for NIHTB Scores for Combined Stroke Sample (n = 131)*  

NIHTB Scores

AVLT

BVMT

WRAT

PPVT

WAIS CD

WAIS SS

WAIS LN

DKEFS

PS

0.45

0.62

0.29

0.36

0.50

0.54

0.38

0.30

OR

0.32

0.54

0.88

0.77

0.47

0.48

0.66

0.48

PV

0.40

0.61

0.74

0.87

0.47

0.52

0.57

0.48

PC

0.28 (0.23)

0.38 (0.36)

0.25 (0.24)

0.38 (0.38)

0.59 (0.57)

0.67 (0.66)

0.35 (0.32)

0.47 (0.44)

LS

0.40

0.60

0.61

0.58

0.50 (0.49)

0.54 (0.54)

0.64

0.47

DCCS

0.45 (0.41)

0.57 (0.55)

0.51 (0.50)

0.55 (0.55)

0.60 (0.57)

0.65 (0.64)

0.49 (0.47)

0.54 (0.51)

Flank-er

0.33 (0.28)

0.50 (0.48)

0.37 (0.36) 

0.50 (0.50)

0.61 (0.58)

0.61 (0.59)

0.41 (0.38)

0.46 (0.43)

*All correlations p < .01

Note: NIHTB scores: PS = Picture sequencing; OR = Oral reading recognition; PV = Picture vocabulary; PC = Pattern comparison; LS = List sorting; DCCS = Dimensional Change Card Sort; Flanker = Flanker inhibitory control and attention; AVLT = Auditory Verbal Learning Test (Rey); BVMT = Brief Visuospatial Memory Test-Revised; WRAT = Word Reading Achievement Test; PPVT = Peabody Picture Vocabulary Test, Fourth Edition; WAIS CD = Wechsler Adult Intelligence Scale – Digit Symbol Coding; WAIS SS =  Wechsler Adult Intelligence Scale – Symbol Search;   WAIS LN = Wechsler Adult Intelligence Scale – Letter Number Sequencing, Fourth Edition; DKEFS = Delis Kaplan Executive Functioning System

• NIHTB scores discriminated between mild and moderate/severe stroke (see Normative Data for mean T scores):
  • Individuals with moderate/severe stroke performed significantly worse on the Fluid composite score and subtests: Picture sequence memory test, Pattern comparison, Flanker, and DCCS (p <0.01)
  • Persons with moderate/sever stroke also performed significantly worse on the Crystallized composite score and subtest for Oral reading recognition (p < 0.05)

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Stroke: (Carlozzi et al, 2017a)

Effect sizes (Cohen’s d) by stroke severity for NIHTB – Cognition Battery composite and subtest scores

NIHTB scores

Mild stroke

Moderate/severe stroke

Composite scores

   Fluid

-0.64

-1.64

   Crystallized

0.05

-0.41

Subtest scores

   Picture vocabulary

0.06

-0.27

   Oral reading recognition

0.01

-0.46

   Picture sequence memory test

-0.36

-0.98

   Pattern comparison

-0.46

-1.25

   List sorting

-0.42

-0.75

   Flanker

-0.50

-1.25

   DCCS

-0.57

-1.18

Brain Injury: (Tulskey et al, 2017; n = 182 (mild or moderate, n = 83 or severe, n = 99); mean age = 38.6 (17.4) years; mean time since TBI = 5.8 years (5.6) years).

• 16th percentile (T score ≤ 40) is the cutoff for a low score. Complicated mild/moderate TBI and severe TBI respectively had individuals with 59% and 75% of individuals have one or more low scores.  Normative base rate of 46% suggests that .54 specificity is poor. 

Brain Injury: (Tulskey et al, 2017)

National Institute of Health (NIH) Toolbox (NIHTB) Demographically Adjusted T  Scores for Individuals with Complicated Mild/Moderate TBI, Severe TBI, and Matched Controls 

NIHTB scores

Comp mild/moderate TBI

Mean (SD)

(n = 74)

Severe TBI

Mean (SD)

(n = 84)

Control

Mean (SD)

(n = 158) 

Composite scores*

   Fluid

92.2 (18.7)

82.4 (18.4)

101.6 (16.0)

   Crystallized

103.2 (15.9)

96.2 (14.7)

101.7 (15.2)

Subtest scores

   Picture vocabulary

102.9 (15.4)

97.1 (14.2)

101.5 (15.4)

   Oral reading recognition

103.2 (15.8)

96.4 (15.3)

102.0 (15.5)

   Picture sequence memory test

92.5 (16.2)

84.2 (15.5)

100.2 (14.3)

   Pattern comparison

96.7 (15.9)

89.8 (17.2)

101.8 (15.9)

   List sorting

96.3 (15.5)

91.2 (15.0)

102.1 (15.3)

   Flanker

94.0 (17.2)

86.2 (16.4)

100.8 (14.6)

   DCCS

94.9 (16.7)

90.1 (16.6)

101.1 (15.5)

*Fluid cognitive composite score combines Dimensional Change Card Sort (DCCS) Test, Flanker Test of Executive Function Inhibitory Control and Attention, Picture Sequence Memory Test of episodic memory, List Sorting Working Memory Test, and Pattern Comparison Processing Speed Test. Crystallized cognitive composite score includes Picture Vocabulary and Oral Reading scores.

Brain Injury: (Carlozzi et al, 2017); n = 184; mean age = 39.83 (17.4) years; male = 64.1%; mean time since TBI = 5.95 (5.54) years)

NIHTB Cognition T Scores for TBI Participantsa

NIHTB scores

n

Mean (SD)

% Impairedb

Composite scores

   Fluid

163

41.75 (13.00)

41.1

   Crystallized

163

49.79 (9.54)

16.6

Subtest scores

   Picture vocabulary

177

49.41 (9.60)

15.8

   Oral reading recognition

177

49.86 (10.82)

16.9

   Picture sequence memory test

161

42.35 (10.88)

43.5

   Pattern comparison

161

45.44 (10.86)

28.0

   List sorting

161

44.75 (10.55)

32.3

   Flanker

161

43.21 (11.10)

38.5

   DCCS

161

44.42 (10.49)

33.5

aT scores are demographically adjusted for age, sex, education, and race/ethnicity.

b% impairment reflects individuals with scores >1 SD beyond the mean on a given test in the negative direction.

Convergent validity: 

Brain Injury: (Tulskey et al., 2017)

• Excellent correlation between Oral Reading Recognition and Wide Range Achievement Test, 4th Edition (r = 0.83)
• Excellent correlation between Picture Vocabulary and Peabody Picture Vocabulary Test, 4th Edition (r = 0.80)
• Adequate correlation between List Sorting and WAIS-IV Letter Number Sequencing (r = 0.56)
• Excellent correlation between Picture Sequence Memory and Brief Visuospatial Memory Test-Revised (r = 0.68)
• Excellent correlation between Pattern Comparison and WAIS-IV Coding (r = 0.69)
• Adequate correlation between Flanker Inhibitory Control and DKEFS Color-Word Interference-Inhibition (r = -0.46)
• Adequate correlation between Dimensional Change Card Sort and Wisconsin Card Sort Test (r = -0.42)

Discriminant validity:

Repetitive Head Impact (RHI): (Amadon et al., 2023; n = 176, mean age = 21.19 (1.63), female = 60 (34%) (n = 115 in contact sport, mean age = 21.46 (1.65), female = 20 (17%); and 61 in non-contact sport, mean age = 20.67 (1.45), female = 40 (66%))

• Significant discriminate ability across the whole sample of the Picture Sequence Memory Test (episodic memory) to distinguish between contact sport and non-contact sport athletes (F(1,171) = 7.16, p = 0.008), years of exposure (F(1,171) = 4.19, p = 0.042), and extensive vs. non-extensive RHI exposure based on traumatic encephalopathy syndrome (TES) criteria (F(1,171) = 5.12, p = 0.025)
  • A similar—although not significant--effect was observed for age of first exposure (AFE), with athletes with AFE < 12 having better episodic memory than those with AFE 12+/none (F(1,171) = 3.88, p = 0.05)
• Significant discriminate ability among females of the Picture Sequence Memory Test (episodic memory) to distinguish between contact sport and non-contact sport athletes (F(1,56) = 5.05, p = 0.03), years of exposure (F(1,56) = 4.54, p = 0.04), athletes with AFE < 12 vs. those with AFE 12+/none (F(1,56) = 4.76, p = 0.03), and extensive vs. non-extensive RHI exposure based on TES criteria (F(1,56) = 4.99, p = 0.03)
• Although the direction of the relationships was the same as those seen in women, among males there were no significant relationships between RHI measures and episodic memory performance. 

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Mixed populations: (Weintraub et al, 2013b; n = 476, age range = 3-85 years, female = 53.1%, stratified sample of community-dwelling individuals)

• Excellent  test-retest validity for Flanker Inhibitory Control and Attention (ICC = 0.9­6)
• Excellent test-retest validity for Dimensional Change Card Sort (ICC = 0.94)
• Acceptable test-retest validity for List Sorting Working Memory (ICC = 0.89)
• Acceptable  test-retest validity for Pattern Comparison (ICC = 0.82)
• Acceptable test-retest validity for Picture Sequence Memory (ICC = 0.78)
• Excellent  test-retest validity for Picture Vocabulary (ICC = 0.94)
• Excellent test-retest validity for Oral Reading and Recognition (ICC = 0.99)

Convergent validity:

Mixed populations: (Weintraub et al., 2013b, age range = 8-85)

• Adequate  convergent validity of Flanker Inhibitory Control and Attention with WISC-IV/WAIS-IV Letter-Number Sequencing, Coding, Symbol Search averagea (r = -0.48b)
• Adequate  convergent validity of Dimensional Change Card Sort with D-KEFS Inhibition (r = -0.51b)
• Adequate  convergent validity of List Sorting Working Memory with WISC-IV/WAIS-IV Letter-Number Sequencinga/Paced Auditory Serial Addition Test average (r = 0.58b)
• Adequate,  convergent validity of Pattern Comparison with WISC-IV/WAIS-IV Coding/Symbol Search averagea (r = 0.49b)
• Excellent  convergent validity of Picture Sequence Memory with  BVMT-R/Rey Auditory Verbal Learning Test averagea (RALVT) 0.69b)
• Excellent  convergent validity of Picture Vocabulary with PPVT-4 (r = 0.78b)
• Excellent  convergent validity of Oral Reading and Recognition with Wide Range Achievement Test, 4th Editon (WRAT-4) (r = 0.93b)

Discriminant validity:

Mixed populations: (Weintraub et al., 2013b, age range = 8-85) 

• Excellent  discriminant validity of Flanker Inhibitory Control and Attention with PPVT-4 (r = 0.15c)
• Excellent  discriminant validity of Dimensional Change Card Sort with PPVT-4 (r = 0.14d)
• Excellent  discriminant validity of List Sorting Working Memory with PPVT-4 (r = 0.30b)
• Excellent  discriminant validity of Pattern Comparison with PPVT-4 (r = 0.12d)
• Excellent  discriminant validity of Picture Sequence Memory with PPVT-4 (r = -0.08)
• Excellent  discriminant validity of Picture Vocabulary with BVMT-R/RAVLT averagea (r = 0.08)
• Excellent  discriminant validity of Oral Reading and Recognition with BVMT-R/RAVLT averagea (r = 0.19b)

aMeasure used dependent upon subjects’ age.

bp < 0.001

cp < 0.01

dp < 0.05

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Multiple Sclerosis: (Manglani et al., 2022; n = 87; age range = 30-59; subjects diagnosed with relapsing-remitting MS; participating in a randomized control trial comparing effects of a physical activity intervention with a water consumption intervention on cognition)

Unadjusted standard scores on NIH Toolbox Cognition Battery

Cognitive Test

Mean

Std. Dev.

Pattern Comparison

98.5

16.8

List Sorting

104

9.58

Picture Sequence Memory

105

15.0

DCCSa

104

7.79

Flanker

97.5

6.52

Oral Reading

108

5.76

Picture vocabularyb

111

7.86

NIH Fluid Cognition

102

10.5

aDimensional Change Card Sort

bDiscriminant validity measure

Convergent validity:

Multiple Sclerosis: (Manglani et al., 2022)

• Adequate convergent validity of the pattern comparison (Processing Speed) subtest with the WAIS-IV Cancelation Test (mean concordance correlation coefficient (Mccc) = 0.46)
• Adequate convergent validity of the list sorting (Working Memory) subtest with the WAIS-IV Digit Span Test (Mccc = 0.34)
• Adequate convergent validity of the picture sequence (Episodic Memory) subtest with the California Verbal Learning Test, Second Edition (Mccc = 0.33)
• Poor convergent validity of the DCCS and the Flanker Test (Mccc = 0.20) (both Executive Function) with the Delis-Kaplan Executive Function System Sorting Test (DKEFS)

Discriminant validity:

Multiple Sclerosis: (Manglani et al., 2022)

• Excellent discriminant validity of the pattern comparison subtest (Processing Speed) with the Picture Vocabulary Test (concordance correlation coefficient (CCC) = 0.071)
• Excellent discriminant validity of the list sorting (Working Memory) pattern comparison subtest with the Picture Vocabulary Test (CCC = 0.29)
• Excellent discriminant validity of the picture sequence memory subtest with the Picture Vocabulary Test (CCC = 0.19)
• Excellent discriminant validity of the DCCS (CCC = 0.16) and the Flanker Test (CCC = 0.11) (both Executive Function) with the Picture Vocabulary Test

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Pediatric Epilepsy: (Matuska et al., 2024; n = 47; male = 55%; mean age = 10.0 (3.7) years; age range = 4.2-18.3 years; mean age of seizure onset = 4.9 (3.2) years; number of antiseizure medications at testing: 0 = 8.5%, 1 = 36.2%, 2 = 31.9%, 3 = 14.9%, 4 = 8.5%; history of Electrical Status Epilepticus in Sleep (ESES) = 46.8%)

Average performance of pediatric epilepsy patients on two NIH Toolbox Cognition Battery tests (n = 46)

Cognitive Test

Mean (SD)

Range

Flanker Inhibitory

87.2 (15.1)

54-124

Pattern Comparison

81.0 (24.6)

33-163

Concurrent validity:

Pediatric Epilepsy: (Matuska et al., 2024)

• Adequate concurrent validity between Flanker Inhibitory Test and Pattern Comparison Test (r = 0.514, p < 0.001)
• Adequate concurrent validity between Pattern Comparison Test and Test of Everyday Attention for Children (TEA-Ch) (Sky Search – Accuracy) (r = 0.459, p < 0.05)
• Adequate concurrent validity between Flanker Inhibitory Test and Wechsler Intelligence Scale for Children (WISC) (Working Memory Index (WMI) (r = 0.455, p = 0.009)
• Adequate concurrent validity between Pattern Comparison Test and WISC (WMI) (r = 0.381, p = 0.035)
• Adequate concurrent validity between Flanker Inhibitory Test and Semantic Fluency (age-appropriate version of A Developmental NEuroPSYchological Assessment, Second Edition (NEPSY-II) or the Delis-Kaplan Executive Function System (D-KEFS)) (r = 0.373, p = 0.023)
• Adequate concurrent validity between Pattern Comparison Test and Semantic Fluency (age-appropriate version of NEPSY-II or D-KEFS (r = 0.365, p = 0.028)
• Adequate concurrent validity between Flanker Inhibitory Test and Comprehensive Test of Phonological Processing, Second Edition (CTOPP-2) (r = 0.577, p = 0.001)
• Adequate concurrent validity between Flanker Inhibitory Test and Wechsler Intelligence Scale for Children (WISC) (Processing Speed Index (PSI) (r = 0.378, p = 0.030)
• Adequate concurrent validity between Flanker Inhibitory Test and Grooved Pegboard w/dominant hand (r = 0.554, p < 0.001) and non-dominant hand (r = 0.420, p = 0.008)
• Adequate concurrent validity between Flanker Inhibitory Test and Verbal IQ (r = 0.458, p = 0.002)
• Adequate concurrent validity between Flanker Inhibitory Test and Nonverbal IQ (r = 0.495, p = 0.002)
• Adequate concurrent validity between Pattern Comparison Test and Nonverbal IQ (r = 0.483, p = 0.002)
• Agreement statistics conducted between the two NIHTB-CB tasks and the above clinical measures found that overall agreement ranged from 57 to 74% for Flanker and from 52 to 72% for Pattern Comparison.
  • Sensitivity was lower for Flanker (42-67%) than for Pattern Comparison (45-83%)
  • Specificity was higher for Flanker (70-83%) than for Pattern Comparison (48-83%)

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a). 

Intellectual Disability: (Shields et al., 2020; n = 242, mean age = 15.71 (5.15) years (Down syndrome = 91, mean age = 15.88 (5.17) years; Fragile X syndrome = 75, mean age = 16.16 (4.92) years; Other ID = 76, mean age = 15.05 (5.35) years); chronological age from 6 through 25 years; full-scale IQ <80 on Stanford-Binet, 5th edition (SB-5); mental age of at least 3.0 years on the SB-5)  

• Acceptable Total test-retest reliability for Flanker Inhibitory Control and Attention (ICC = 0.74)
• Acceptable test-retest reliability for Dimensional Change Card Sort (ICC = 0.71)
• Acceptable Total test-retest reliability for List Sorting Working Memory (ICC = 0.74)
• Acceptable Total test-retest reliability for Pattern Comparison (ICC = 0.77)
• Poor Total test-retest reliability for Picture Sequence Memory (ICC = 0.47 with form A-A and ICC = 0.55 with form A-B)
• Acceptable Total test-retest reliability for Picture Vocabulary (ICC = 0.85)
• Excellent Total test-retest reliability for Oral Reading and Recognition (ICC = 0.96)
• Acceptable Total test-retest reliability for Fluid Composite (ICC = 0.83)
• Excellent Total test-retest reliability for Crystalized Composite (ICC = 0.93)
• Excellent Total test-retest reliability for Cognitive Function Composite (ICC = 0.92)

Intellectual Disability: (Hessl et al., 2016; Fragile X Syndrome: n = 63, mean age = 19.3 (8.3) years, mean mental age = 5.3 (1.6) years )

• Acceptable test-retest validity for Flanker Inhibitory Control and Attention (ICC = 0.75)
• Acceptable test-retest validity for Dimensional Change Card Sort (ICC = 0.88)
• Acceptable test-retest validity for List Sorting Working Memory (ICC = 0.84)
• Excellent test-retest validity for Pattern Comparison (ICC = 0.90)
• Acceptable test-retest validity for Picture Sequence Memory (ICC = 0.76)
• Acceptable test-retest validity for Picture Vocabulary (ICC = 0.77)
• Excellent test-retest validity Oral Reading and Recognition (ICC = 0.99)

Convergent validity:

Intellectual Disability: (Shields et al., 2020)

• Adequate convergent validity of Flanker Inhibitory Control and Attention with Conners Kiddie Continuous Performance Test, 2nd Edition (r = -0.52*)
• Adequate convergent validity of Dimensional Change Card Sort with NEPSY Inhibition (NEPSY-IN) subtest (r = 0.48*)
• Excellent convergent validity of List Sorting Working Memory with Stanford-Binet, 5th Edition (SB-5) Verbal Working Memory (r = 0.65*)
• Excellent convergent validity of Pattern Comparison with Wechsler Preschool and Primary Scale of Intelligence, 4th Edition Bug Search (r = 0.66*)
• Adequate convergent validity of Picture Sequence Memory with Leiter International Performance Scale, 3rd Edition Forward Memory (Leiter-FM)(r = 0.47*)
• Excellent convergent validity of Picture Vocabulary Test with Peabody Picture Vocabulary Test, 4th Edition (r = 0.83*)
• Excellent convergent validity of Oral Reading and Recognition with Woodcock Johnson 4th Edition Letter-Word Identification (WJ-LW) (r = 0.92*)
• Excellent convergent validity of Fluid Composite with SB-5 Fluid Reasoning IQ (r = 0.60*)
• Excellent convergent validity of Crystalized Composite with SB-5 Verbal IQ (r = 0.75*)

*Significant at p < 0.001

Discriminant validity:

Intellectual Disability: (Shields, et al, 2020)

• Adequate discriminant validity of Flanker Inhibitory Control and Attention with WJ-LW (r = -0.53*)
• Adequate discriminant validity of Dimensional Change Card Sort with WJ-LW (r = 0.36*)
• Adequate discriminant validity of List Sorting Working Memory with WJ-LW (r = 0.49*)
• Adequate discriminant validity of Pattern Comparison with WJ-LW (r = 0.45*)
• Adequate discriminant validity of Picture Sequence Memory with WJ-LW (r = 0.50*)
• Adequate discriminant validity of Picture Vocabulary Test with Leiter-FM (r = 0.47*)
• Adequate discriminant validity of Oral Reading and Recognition with Leiter-FM (r = 0.58*)
• Poor discriminant validity of Fluid Composite with SB-5 Verbal IQ (r = 0.61*)
• Poor discriminant validity of Crystalized Composite with SB-5 Fluid Reasoning IQ (r = 0.68*)

*Significant at p < 0.001

The Cognition Battery (CB) team selected the subdomains of attention, executive function, episodic memory, language, working memory, and processing speed based on two Requests for Information (RFIs) that were solicited online to obtain the input from 293 experts. The RFIs were then followed-up by telephone interviews of a subset of 44 experts. After selecting the specific tests for the subdomains, the NIH Toolbox was presented to an expert advisory panel who provided written critiques of the subdomains and instruments that were reviewed and addressed by the CB team. Lastly, several conference calls were conducted with 16 expert consultants to present the version of the CB created for validation testing and to invite feedback prior to initiating the validation study (Weintraub et al., 2013a).