Task 4. Andrew’s problems become clear
Learning goals
What does evidence based psychological assessment entail?
Bornstein, R. F. (2017). Evidence-based psychological assessment, 435-445.
Abstract
In recent years there has been increasing emphasis on evidence-based practice in
psychology (EBPP), and in most health care professions the primary focus of EBPP has been
on treatment. Comparatively little attention has been devoted to applying the principles of
EBPP to psychological assessment, despite the fact that assessment plays a central role in
numerous domains of empirical and applied psychology (e.g. research, forensics, behavioral
health, risk management, diagnosis and classification in mental health settings,
documentation of neuropsychological impairment and recovery, personnel selection and
placement in organizational contexts). This article outlines the central elements of evidence-
based psychological assessment (EBPA), using the American Psychological Association’s
definition of EBPP as integration of the best available research with clinical expertise in the
context of patient characteristics, culture, and preferences. The integration of patient
characteristics, culture, and preferences is discussed in terms of the complex interaction of
patient and assessor identities and values throughout the assessment process.
Introduction
Evidence-based practice (EBP) focusses on the importance of rigorous empirical verification
of medical interventions to maximize the impact of health care expenditures. EBP guidelines
have been developed for a number of health and associated health disciplines. The American
Psychological Association defines evidence-based practice in psychology (EBPP) as
“integration of the best available research with clinical expertise in the context of patient
characteristics, culture and preferences.
Historically the emphasis in EBPP has been on treatment, with respect to refining diagnosis
or evaluating the effectiveness of intervention. However, Psychological assessment plays a
central role in multiple domains. Given the broad application of assessment in contemporary
practice and research, formal guidelines for evidence-based psychological assessment
(EBPA) are being developed.
This article represents a preliminary set of principles and strategies intended to stimulate
dialogue and encourage continued attention to this issue. EBPA is discussed with respect to
three components of EBPP: (1) research; (2) clinical expertise; and (3) patient characteristics,
culture, and preferences.
Best available research (1)
Psychological testing is a relatively straightforward process wherein a particular test is
administered to obtain a specific score. Then a descriptive meaning can be applied to the
score based on normative, nomothetic findings. In contrast, psychological assessment is
concerned with the clinician who takes a variety of test scores, generally obtained from
multiple test methods, and considers the data in the context of history, referral information,
and observed behavior to understand the person being evaluated, to answer the referral
,questions, and then to communicate findings to the patient, his or her significant others, and
referral sources. Because of the emphasis on testing rather than assessment, the Standards
for Educational and Psychological Testing provides only minimal guidance in this area. It says
little about procedures and strategies for psychological assessment (the process of
integrating data from various psychological tests that often yield different results, and of
integrating psychological test data with evidence from other sources). The International Test
Commission Guidelines on Test Use (ITC) devotes greater attention to issues regarding
psychological assessment, as well as psychological testing. However, neither discusses the
psychological processes that underlie patients’ responses to different types of test items and
tasks, considers the interpersonal dynamics of the assessment process, or discusses how
competence and proficiency in psychological assessment could be operationalized and
evaluated.
Best available research: testing
Contemporary unified models of validity include these (and other) standard psychometrics,
as well as considering the real-world consequences of test use (impact, fairness, etc.). As
Messick noted, although they can be described separately, in practice it is not possible to
isolate the psychometric properties of a measure from the impact that scores derived from
this measure have in different contexts and settings (termed “consequential validity”).
Bornstein argued that to obtain a picture as complete as possible of a test’s psychometric
soundness and consequential validity, traditional outcome-focused validity assessment must
be complemented by process-focused validity assessment. Process-focused assessment of
test score validity has implications for optimizing the utility of test scores, but also for
understanding test bias, and the ways in which test batteries could be tailored to correspond
more effectively with patient characteristics, culture, and preferences.
In a process-based framework the array of assessment tools is classified, grouping the
instruments into 6 categories based on mental activities and behaviors involved in
responding to the tests.
Inherent in the process-focused approach is an extended definition of test bias: whereas
Messick’s consequential validity model conceptualizes bias in terms of differential impact of
test scores in different groups, the process-focused model argues that test bias can be
traced in part to differential psychological processes engaged by similar test items in
different groups. For example, items will engage different psychological processes in women
and men. In a narcissism test items tend to evoke greater defensiveness in women, and in a
dependency test items evoking greater defensiveness in men. Studies documenting the
, impact of stereotype threat on psychological test results represent additional examples of
process-based bias (e.g., when the implications of incorrect answers to intelligence test
items are interpreted differently by groups, resulting in different emotional reactions to
items).
Best available research: assessment
Validating psychological assessment results is more challenging than validating psychological
test scores, as few norms exist regarding what constitutes acceptable validity evidence for
psychological assessments. A well-constructed assessment battery must include tests that
yield valid scores for the purpose of the tests. The combination of tests that comprise the
battery is salient as well: Each test must add incremental (step-wise) validity to the battery,
contributing unique information that other tests do not provide. In addition, test results
must be combined in a meaningful way (with attention to both outcome and process), and
these integrated test results must be contextualized appropriately with respect to patient
history, referral question, and other salient information. Finally, assessment findings must be
framed and communicated in a useful way to patient, referent, and others.
Operationalizing assessment effectiveness involves a shift in emphasis from validity to
clinical utility. Clinical utility can be quantified in multiple ways, with each method
contributing unique information regarding the effectiveness and impact of assessment
results. Although most patients lack the training and experience of practicing clinicians and
assessment experts, they have a unique perspective that could help fine-tune test batteries
and assessment reports. It helps to resonate more closely with patient values and
preferences, the third pillar of EBPP.
In operationalizing the concept of “best available research evidence” the American
Psychological Association recommended use of multiple avenues of evidence to examine the
validity and clinical utility of an intervention or assessment strategy. Given the contrasting
psychological processes engaged by different tests, it is important to emphasize test score
divergences as well as convergences to maximize the utility of assessment data.
Rendering interpretations at the level of the individual patient is more challenging than
documenting them empirically in clinical or nonclinical samples. From a psychometric
perspective, the reliability of the difference between two scores is lower than the reliability
of the scores themselves. Multiple comparisons increase the probability of Type I errors.
Thus, interpretation of test score divergences must be done with caution, and with attention
to the psychometric constraints associated with these comparisons.
Clinical expertise (2)
The use of Bayesian techniques and regression strategies helps optimize accuracy in
assessment and treatment planning, and minimize misdiagnosis and missed diagnosis.
Although effective use of actuarial methods represents an important component of EBPA,
these methods yield optimal results when they are implemented by a clinician with training
and experience in psychological assessment. With this in mind, the second pillar (clinical
expertise) in the American Psychological Association’s definition of EBPP is problematic. The
American Psychological Association’s use of the term expertise is not in line with prevailing
views, as they refer to competence attained by psychologists through education, training,
and experience that results in effective practice. The term is not meant to refer to
extraordinary performance that might characterize an elite group of clinicians.
