Adapted Fresno Test

Last Updated

November 14, 2018

Purpose

The Adapted Fresno Test (AFT) is a change made to the original FT to include scenarios relevant to rehabilitation professionals and remove some statistical questions. The AFT examines a participant’s ability to write a population, intervention, comparison, outcome (PICO) formula question, document a search strategy, identify an appropriate study design to best answer a clinical question, discuss the interpretation and validity of research studies, and develop parameters to judge the clinical and statistical significance of the findings.

Please see the original Modified Fresno Test and Fresno Test.

Link to Instrument

Instrument Details

Acronym AFT

Area of Assessment

Reasoning/Problem Solving

Assessment Type

Other

Administration Mode

Paper & Pencil

Cost

Free

Actual Cost

$0.00

Allied Health Care Professions

Items: Select one of two clinical scenarios to answer seven questions
● Version 1 (clinical scenario A or B); Version 2 (clinical scenario C or D); Version 3 (clinical scenario E or F; follow-up assessment)
Score: 0 to 156
● Four grading categories are used (not evident, limited, strong, and excellent), each corresponding to a specific number of points. For example, a response that does not mention a patient population or that uses an irrelevant or inappropriate descriptor earns 0 points (not evident); use of a single general descriptor constitutes a limited answer (1 point); mentioning one appropriate but not specific descriptor is a strong answer (2 points); and using relevant and appropriate descriptors is excellent (3 points). Each criterion is scored according to these categories, and the sum of points for all criteria is the score for that item.
Eight-page scoring rubric available from the authors upon request
Researchers and educators using the AFT are expected to provide rater training and evaluate rater agreement against results presented in the study

Number of Items

7 questions

Paper or electronic access to the test

20 minutes

Time to grade: 20 minutes (McCluskey & Lovarini, 2005; McCluskey & Bishop, 2009)

Required Training

No Training

Instrument Reviewers

Kathy Sanders, MS, OTR/L and Andrea Vassev, MPT

Considerations

Please see the original Modified Fresno Test and Fresno Test.
The FT, AFT, and MFT are best utilized for assessing participants’ knowledge of EBM or EBP. It is important to note that they may not test an individual’s ability to successfully apply EBM or EBP in the clinical setting (Ramos et al., 2003).
It is essential to select the correct Fresno Test for your discipline because the case scenarios and test questions are geared toward specific disciplines. For example, an OT or PT would not be appropriate for the FT as its clinical case scenarios are specific to medical practice.
The FT, AFT, and MFT can be used when measuring a change in knowledge after EBP instruction or to determine areas of weakness before an instruction or practice is implemented (Argimon-Pallàs et al., 2011). It may be helpful to compare groups based on pretest upper and lower (25%) quartiles if making comparisons utilizing a pre and posttest design (McCluskey & Bishop, 2009).
Consider giving the pre and posttest at least four weeks apart, to help prevent bias based on test recall (Argimon-Pallàs et al., 2011).
MFT raters participated in three different training sessions on three different days; 2-hour introduction on the test, scoring rubric, and standardized data collection form; 2.5 hours of practice with the scoring rubric; and 1.5 hours for questions and discussion (Tilson, 2010).
Information concerning rater training in the clinical setting is lacking, therefore it is unclear how long the training process would take and also what support and resources may be needed when utilizing novice raters.
There are two additional studies that are adaptations of the original Fresno Test, specific to pharmacy students (Coppenrath, Filosa, Akselrod, & Carey, 2017), and to pediatric bedside nurses (Laibhen-Parkes, Kimble, Melnyk, Sudia, & Codone, 2018), that are not included in this review.

Cut-Off Scores

Occupational Therapists: (McCluskey & Bishop, 2009; n = 106; all female OTs; five years post-graduation 80%; did not hold a postgraduate qualification 68%; employed full-time 77%; baseline pretest score of ≤ 78/156 points classified participant as a low scorer (n = 83) and a baseline pretest score > 78/156 points classified participant as a high scorer (n = 23); posttest AFT occurred after 2-day EBP workshop)

Based on the authors’ expert opinion, participants scoring ≤ 78/156 points were classified as low scorers and participants scoring > 78/156 points classified as high scorers.

Normative Data

Occupational Therapists: (McCluskey & Bishop, 2009; following EBP training; n = 106; mean score ± SD)

Total (0-156)	57.7 (27.3)	78.3 (18.6)
Low scorers – participants scoring below 78/156 preworkshop (n=83)
Q1	7.0 (2.9)	9.7 (1.6)
Q2	13.6 (6.4)	15.2 (4.3)
Q3	3.6 (5.6)	11.9 (4.7)
Q4	7.9 (6.8)	13.4 (6.1)
Q5	6.3 (5.3)	4.8 (4.3)
Q6	7.5 (7.2)	10.6 (7.3)
Q7	1.2 (2.5)	9.1 (9.1)
Total	47.4 (19.8)	74.1 (17.7)
High scorers– participants scoring 78/156 or higher preworkshop (n=23)
Q1	9.5 (1.9)	10.2 (1.5)
Q2	20.9 (2.8)	18.6 (2.9)
Q3	12.7 (7.2)	13.3 (3.8)
Q4	15.1 (5.2)	18.8 (3.8)
Q5	12.3 (4.6)	6.9 (4.8)
Q6	18.1 (5.3)	14.9 (7.3)
Q7	6.6 (5.9)	10.7 (5.0)
Total	95.2 (14.4)	93.3 (13.7)

Interrater/Intrarater Reliability

Occupational Therapists: (McCluskey & Bishop, 2009; raters were two OT lecturers with expert EBP knowledge)

Excellent inter-rater reliability; test version 1 (n = 10) (ICC = 0.96)
Excellent inter-rater reliability; test version 2 (n = 10) (ICC = 0.91)

AFT rater training consisted of a 2-hour training session utilizing scored and unscored copies of the test, after which raters had two weeks to practice rating 20 tests with the scoring matrix (McCluskey & Bishop, 2009).

Occupational Therapists and Physical Therapists: (Lizarondo et al.,2013; n = 55; OTs and PTs; bachelor’s degree 62%; completed postgraduate degree 38%; less than 50% had prior EBP training or participation in research; participants took the AFT one time; raters consisted of four PTs with varying levels of professional experience; Rater 1 - master’s in sport and musculoskeletal PT, two years outpatient and hospital experience, one year EBP-related research experience, no teaching experience or other professional qualifications; Rater 2 - master’s in manual and sport PT, five years outpatient experience, 2.5 years EBP-related research experience, occasional [clinical demonstration] teaching experience, level 2 sports trainer certification; Rater 3 - PhD candidate and health-related master’s in PT and clinical psychology, internship experience only, seven years EBP-related research experience, 16 year undergraduate and one year postgraduate teaching experience; Rater 4 - PhD candidate and master’s in PT, two years hospital experience, 8.5 years EBP-related research experience, 14 years undergraduate and nine years postgraduate teaching experience, director of research center for five years)

Reliability Estimates for Inexperienced and Experienced raters

	ICC (95% CI)
Adapted Fresno Test item	Inexperienced^*	Experienced^†
2. Where might you find answers to these questions? Name as many possible sources of information as you can. List advantages and disadvantages.	0.96 (0.86–0.99)	0.97 (0.86–0.99)
3. What type of study (design) would best answer your clinical question and why?	0	0.94 (0.80–0.98)
4. Describe the search strategy you might use in Medline topics, fields, rationale and limits.	0.66 (−0.04 to 0.90)	0.70 (0.06–0.91)
5. What characteristics of a study determine if it is relevant?	0.24 (−0.34 to 0.70)	0.95 (0.83–0.99)
6. What characteristics of a study determine its validity?	0.70 (−0.02 to 0.91)	0.98 (0.92–0.99)
Total score	0.58 (−0.26 to 0.88)	0.92 (0.72–0.98)

^*Inexperienced: Raters 1 and 2.

^†Experienced: Raters 3 and 4.

Dieticians/Nutritionists, Social Workers, and Speech Pathologists: (Lizarondo et al., 2014; speech pathologists (SP), n = 10; social workers (SW), n = 16; dietician/nutritionists (DN), n = 12; bachelor’s degree 50%; postgraduate degree 50%; previous EBP training or research < 50%; majority in clinical practice more than 10 years; participants took the AFT one time; raters were four individuals experienced in research and teaching EBP to allied health students)

Excellent SP inter-rater reliability (ICC = 0.93)
Excellent SW inter-rater reliability (ICC = 0.83)
Excellent DN inter-rater reliability (ICC = 0.92)

Internal Consistency

Occupational Therapists:

Adequate to Excellent internal consistency (Cronbach’s alpha = 0.72 to 0.84) (McCluskey & Lovarini, 2005; pretest with version 1, n = 114; posttest after 2-day EBP workshop with version 2, n = 106; eight month follow-up with version 3, n = 51; majority of participants held an undergraduate degree in OT)
Adequate internal consistency (Cronbach’s alpha = 0.74) (McCluskey & Bishop, 2009)

Speech pathologists (Lizarondo et al., 2014)

Adequate internal consistency (Cronbach’s alpha = 0.71)

Social workers (Lizarondo et al., 2014)

Poor internal consistency (Cronbach’s alpha = 0.68)

Dieticians (Lizarondo et al., 2014)

Adequate DN internal consistency (Cronbach’s alpha = 0.74)

Content Validity

Occupational Therapists: (McCluskey & Lovarini, 2005)

Author McCluskey reduced the original FT questions from 12 to seven and included new clinical scenarios relevant to OTs. Questions pertaining to diagnosis or complex statistical calculations were removed.
Six new clinical scenarios were included, two each for versions 1, 2, and 3, to minimize practice effects.
The seven questions in the AFT were scored using standardized grading criteria, similar to those reported by Ramos et al. (2003).

Face Validity

Occupational Therapists: (McCluskey & Bishop, 2009)

Due to low interrater reliability on certain subtest items in the McCluskey and Lovarini (2005) study, original FT questions were further reduced from 12 to seven. Pilot testing with six OTs and two PTs, with a range of experience and knowledge about EBP, assisted in question selection. Questions were removed that pertained to diagnosis or complex statistical calculations.
Six new clinical scenarios were written, two each for versions 1, 2, and 3, to minimize practice effects.
Scoring matrix was revised to include examples of “excellent,” “strong,” and “limited” responses, in addition to “not evident” responses that receive no score.

Dieticians/Nutritionists, Social Workers, and Speech Pathologists: (Lizarondo et al., 2014)

Test modifications for the AFT were established through formal feedback from an expert panel consisting of four practitioners from each discipline. The panel represented practitioners with more than 10 years of clinical experience and previous exposure to EBP training or research. The majority had graduate degrees.
New clinical scenarios were developed for each discipline. The scoring rubric of the AFT was considered applicable to the new versions except for questions 1, 2, and 4 that were modified slightly to include additional PICO terms or synonyms.

Responsiveness

Occupational Therapists: (McCluskey & Bishop, 2009)

Large effect size = 0.8

Question	Mean Change (95% Confidence Interval)	Effect Size*
Q1 (0-12)	2.3	(1.7 to 2.9)	0.8
Q2 (0-24)	0.7	(-0.5 to 1.9)	0.1
Q3 (0-24)	6.8	(5.2 to 8.3)	0.9
Q4 (0-24)	5.1	(3.6 to 6.5)	0.7
Q5 (0-24)	-2.4	(-3.6 to -1.1)	-0.4
Q6 (0-24)	1.7	(-0.2 to 3.6)	0.2
Q7 (0-24)	7.1	(5.3 to 8.8)	1.7
Total (0-156)	20.6	(15.6 to 25.5)	0.8
Q1	2.8	(2.1 to 3.6)	0.9
Q2	1.6	(0.1 to 3.0)	0.3
Q3	8.4	(6.8 to 10.0)	1.5
Q4	5.5	(3.8 to 7.1)	0.8
Q5	-1.5	(-2.9 to -0.1)	-0.3
Q6	3.1	(0.9 to 5.3)	0.4
Q7	7.9	(5.9 to 9.9)	3.2
Total	26.8	(21.6 to 31.9)	1.4
Q1	0.7	(-0.5 to 1.8)	-0.4
Q2	-2.4	(3.9 to -0.8)	0.9
Q3	0.7	(-2.7 to 3.9)	-0.1
Q4	3.7	(0.4 to 7.1)	-0.7
Q5	-5.4	(-7.9 to -2.9)	0.6
Q6	-7.4	(-10.5 to -4.2)	1.4
Q7	4.1	(1.3 to 6.9)	-0.7
Total	1.8	(-6.4 to 10.1)	0.1

*Effect size calculated with the use of mean change scores divided by the standard deviation of preworkshop score.

Bibliography

Coppenrath, V., Filosa, L. A., et al. (2017). “Adaptation and validation of the Fresno Test of competence in evidence-based medicine in doctor of pharmacy students.” Am J of Pharm Educ 81(6): 1-9. Find it on PubMed

Laibhen-Parkes, N., Kimble, L. P., et al. (2018). “An adaptation of the original Fresno Test to measure evidence-based practice competence in pediatric bedside nurses.” Worldviews Evid Based Nurs 15(3): 230-240. Find it on Wiley Online Library

Lizarondo, L. M., Grimmer, K., et al. (2013). “Interrater reliability of the Adapted Fresno Test across multiple raters.” Physiother 65(2): 135-140. Find it on PubMed

Lizarondo, L. M., Grimmer, K., et al. (2014). “The Adapted Fresno Test for speech pathologists, social workers, and dieticians/nutritionists: Validation and reliability testing.” J Multidiscip Healthc 7: 129-135. Find it on PubMed

McCluskey, A., Bishop, B. (2009). “The Adapted Fresno Test of competence in evidence-based practice.” J Contin Educ Health Prof 29(2): 119-126. Find in on Wiley Online Library

McCluskey, A., & Lovarini, M. (2005). “Providing education on evidence-based practice improved knowledge but did not change behaviour: A before and after study.” BMC Med Educ 5(1): 1-12. Find in on PubMed.