Summary Research Methods in Clinical Neuropsychology
Choosing the study subjects: specification, sampling, and recruitment – Hulley, Newman &
Cummings (2013)
A good choice of study subjects serves the vital purpose of ensuring that the findings in the study
accurately represent what is going on in the population of interest. The protocol must specify a sample
of subjects that can be studied at an acceptable cost in time and money (modest in size and convenient
to access), yet large enough to control random error and representative enough to allow generalizing
study findings to populations of interest. An important precept here is that generalizability is rarely a
simple yes-or-no matter: it is a complex qualitative judgment that depends on the investigator’s choice
of population and of sampling design.
A population is a complete set of people with specified characteristics, and a sample is a subset of the
population.
− Clinical and demographic characteristics define the target population: the large set of people
throughout the world to which the results may be generalized (e.g., teenagers with asthma)
− The accessible population is a geographically and temporally defined subset of the target
population that is available for study (e.g., teenagers with asthma living in the investigator’s
town this year)
− The intended study sample is the subset of the accessible population that the investigator seeks
to include in the study
− The actual study sample is the group of subjects that does participate in the study
The classic Framingham Study was an early approach to scientifically designing a study to allow
inferences from findings observed in a sample to be applied to a population.
In general, analytic studies and clinical trials that address biologic relationships produce more widely
generalizable results across diverse populations than descriptive studies that address distributions of
characteristics.
Inclusion criteria define the main characteristics of the target population that pertain to the research
question. Inclusion criteria that address the geographic and temporal characteristics of the accessible
population often involve trade-offs between scientific and practical goals. On these and other decisions
about inclusion criteria, there is no single course of action that is clearly right or wrong: the important
thing is to make decisions that are sensible, that can be used consistently throughout the study, and
that can be clearly described to others who will be deciding to whom the published conclusions apply.
Effect modification is also called an interaction. The number needed to statistically test for the
presence of effect modification is generally large, and most studies are not powered to detect effect
modification.
1
,Exclusion criteria indicate individuals who meet the inclusion criteria and would be suitable for the
study were it not for characteristics that might interfere with the success of follow-up efforts, the
quality of the data, or the acceptability of randomized treatment. Clinical trials differ from
observational studies in being more likely to have exclusions mandated by concern for the safety of an
intervention in certain patients. A good general rule that keeps things simple and preserves the number
of potential study subjects is to have as few exclusion criteria as possible.
True “population-based” samples are difficult and expensive to recruit, but useful for guiding public
health and clinical practice in the community. One of the largest and best examples is the National
Health and Nutrition Examination Survey (NHANES), a representative sample of U.S. residents.
The size and diversity of a sample can be increased by collaborating with colleagues in other cities, or
by using preexisting data sets such as NHANES and Medicare data. Electronically accessible data sets
from public health agencies, healthcare providing organizations, and medical insurance companies
have come into widespread use in clinical research and may be more representative of national
populations and less time-consuming than other possibilities.
There are 2 types of sampling:
1. Nonprobability samples
− Convenience sample: the study sample is made up of people who meet the entry
criteria and are easily accessible to the investigator. It has obvious advantages in cost
and logistics, and is a good choice for some research questions. The validity of
drawing inferences from any sample is the premise that, for the purpose of answering
the research question at hand, it sufficiently represents the accessible population. With
convenience samples this requires a subjective judgment
− Consecutive sample: consecutively selecting subjects who meet the entry criteria. This
minimizes volunteerism and other selection biases. This approach is especially
desirable, for example, when it amounts to taking the entire accessible population over
a long enough period to include seasonal variations or other temporal changes that are
important to the research question
2
, 2. Probability samples: uses a random process to guarantee that each unit of the population has a
specified change of being included in the sample. This is the gold standard for ensuring
generalizability. It is a scientific approach that provides a rigorous basis for estimating the
fidelity with which phenomena observed in the sample represent those in the population, and
for computing statistical significance and confidence intervals
− Simple random sample: is drawn by enumerating (listing) all the people in the
population from which the sample will be drawn, and selecting a subset at random.
The most common use of this approach in clinical research is when the investigator
wishes to select a representative subset from a population that is larger than she needs
− Systematic sample: resembles a simple random sample in the first step, enumerating
the population, but differs in that the sample is selected by a preordained periodic
process. Systematic sampling is susceptible to errors caused by natural periodicities in
the population, and it allows the investigator to predict and perhaps manipulate those
who will be in the sample. It offers no logistic advantages over simple random
sampling, and in clinical research it is rarely a better choice
− Stratified random sample: begins by dividing the population into subgroups according
to characteristics such as sex or race, and taking a random sample from each of these
“strata.” The Stratified subsamples can be weighted to draw disproportionately from
subgroups that are less common in the population but of special interest to the
investigator.
− Cluster sample: a random sample of natural groupings (clusters) of individuals in the
population. Cluster sampling is useful when the population is widely dispersed and it
is impractical to list and sample from all its elements. A disadvantage of cluster
sampling is the fact that naturally occurring groups are often more homogeneous for
the variables of interest than the population. This means that the effective sample size
(after adjusting for within-cluster uniformity) will be somewhat smaller than the
number of subjects, and that statistical analysis must take the clustering into account
The decision about whether the proposed sampling design is satisfactory requires that the investigator
make a judgment: for the research question at hand, will the conclusions drawn from observations in
the study sample be similar to the conclusions that would result from studying a true probability
sample of the accessible population? And beyond that, will the conclusions be appropriate for the
target population?
An important factor to consider in choosing the accessible population and sampling approach is the
feasibility of recruiting study participants. There are 2 main goals:
1. To recruit a sample that adequately represents the target population, minimizing the prospect
of getting the wrong answer to the research question due to systematic error (bias)
2. To recruit a sufficient sample size to minimize the prospect of getting the wrong answer due to
random error (chance)
The approach to recruiting a representative sample begins in the design phase with wise decisions
about choosing target and accessible populations, and approaches to sampling. It ends with
implementation, guarding against errors in applying the entry criteria to prospective study participants,
and enhancing successful strategies as the study progresses.
A particular concern (especially for descriptive studies) is the problem of nonresponse. The proportion
of subjects selected for the study who consent to be enrolled (the response rate) influences the validity
of inferring that the enrolled sample represents the population. People who are difficult to reach and
those who refuse to participate once they are contacted tend to be different from people who do enroll.
The level of nonresponse that will compromise the generalizability of the study depends on the nature
of the research question and on the reasons for not responding. A nonresponse rate of 25% (a good
achievement in many settings) can seriously distort the estimate of the prevalence of a disease when
the disease itself is a cause of nonresponse.
3
, The degree to which nonresponse bias may influence the conclusions of a descriptive study can
sometimes be estimated during the study by acquiring additional information on a sample of
nonrespondents. The best way to deal with nonresponse bias, however, is to minimize the number of
nonrespondents. The problem of failure to make contact with individuals who have been chosen for
the sample can be reduced by designing a series of repeated contact attempts using various methods
(mail, e-mail, telephone, home visit). Among those contacted, refusal to participate can be minimized
by improving the efficiency and attractiveness of the study, by choosing a design that avoids invasive
and uncomfortable tests, by using brochures and individual discussion to allay anxiety and discomfort,
by providing incentives such as reimbursing the costs of transportation and providing the results of
tests, and by circumventing language barriers with bilingual staff and translated questionnaires.
Falling short in the rate of recruitment is one of the commonest problems in clinical research. The
approaches to this problem are to estimate the magnitude of the recruitment problem empirically with
a pretest, to plan the study with an accessible population that is larger than believed necessary, and to
make contingency plans should the need arise for additional subjects. While recruitment is ongoing it
is important to closely monitor progress in meeting the recruitment goals and tabulate reasons for
falling short of the goals. Understanding why potential subjects are lost to the study at various stages
can lead to strategies for reducing these losses.
It may be helpful to prepare for recruitment by getting the support of important organizations.
Planning the measurements: precision, accuracy, and validity – Hulley, Newman & Cummings
(2013)
Measurements describe phenomena in terms that can be analyzed statistically, and the validity of a
study depends on how well the variables designed for the study represent the phenomena of interest.
Numeric variables can be quantified with a number that expresses how much or how many. There are
2 types of numeric variables:
1. Continuous variables: quantify how much on an infinite scale (e.g., body weight). Continuous
variables are rich in information
2. Discrete numeric variables: quantify how many on a scale with fixed units, usually integers
(e.g., the number of times a woman has been pregnant). Discrete variables that have a
considerable number of possible values can resemble continuous variables in statistical
analyses and be equivalent for the purpose of designing measurements
Phenomena that are not suitable for quantification are measured by classifying them in categories.
There are different types of categorical variables:
1. Dichotomous variables: categorical variables with 2 possible values (e.g., dead or alive)
2. Polychotomous variables: categorical variables with more than 2 categories. These can be
further characterized according to the type of information they contain:
− Nominal variables: have categories that are not ordered (e.g., blood type). Nominal
variables tend to have an absolute qualitative character that makes them
straightforward to measure
− Ordinal variables: the categories do have an order (e.g., severe, moderate, and mild
pain). The additional information is an advantage over nominal variables, but because
ordinal variables do not specify a numerical or uniform difference between one
category and the next, the information content is less than that of discrete or
continuous numeric variables
A good general rule is to prefer continuous over categorical variables when there is a choice, because
the additional information they contain improves statistical efficiency. The continuous variable
contains more information, and the result is a study with more power and/or a smaller sample size.
Continuous variables also allow for more flexibility than categorical variables in fitting the data to the
nature of the variable or the shape of the association, especially when the relationship might have a
complex pattern.
4
