Research Design in Social Research
David de Vaus (summary by Jennie Weemhoff)
Part I: What is research design?
1: The context of design
Description and explanation
Social researchers ask two fundamental types of research questions:
1. What is going on (descriptive research)?
2. Why is it going on (explanatory research)?
Good description provokes the ‘why’ questions of explanatory research.
Answering the ‘why’ questions involves developing causal explanations. Causal explanations
argue that phenomenon Y (e.g. income level) is affected by factor X (e.g. gender).
Direct causal relationship Gender Income level
Indirect causal relationship Gender Education Income level
Good prediction does not depend on causal relationships. Nor does the ability to predict accurately
demonstrate anything about causality. While we can observe correlation, we cannot observe
cause. We have to infer cause.
Deterministic and probabilistic concepts of causation
There are two ways of thinking about causes: deterministically and probabilistically.
Deterministic causation is where variable X is said to cause Y if, and only if, X invariably
produces Y. That is, when X is present then Y will necessarily, inevitably and infallibly occur.
Social sciences is probabilistic rather than deterministic because the complexity of human social
behaviour and the subjective, meaningful and voluntaristic components of human behaviour mean
that it will never be possible to arrive at causal statements of the type ‘if X, and A and B, then Y will
always follow’.
Probabilistic causation is when a given factor increases (or decreases), the probability of a
particular outcome increases. We can improve probabilistic explanations by specifying conditions
under which X is less likely and more likely to affect Y. But we will never achieve complete or
deterministic explanations. Human behaviour is both willed and caused.
1
,Theory testing and theory constructing
Attempts to answer the ‘why’ questions in social science are theories. Theories vary in their
complexity, abstraction and scope. There are two different styles of research: theory testing and
theory building.
Theory building
Theory building is a process in which research begins with observations and uses inductive
reasoning to derive a theory from these observations. The theory is produced after observations
are made, so it is often called post factum theory or ex post facto theorising.
Theory testing
Theory testing begins with a theory and uses theory to guide which observations to make it: it
moves from the general to the particular using deductive reasoning. If the theory is true then
certain things should follow.
What is research design?
The function of a research design is to ensure that the evidence obtained enables us to answer the
initial question as unambiguously as possible. Researchdesign ‘deals with a logical problem and
not a logistical problem’.
Design versus method
Research design is different from the method by which data are collected. Data for any design can
be collected with any data collection method.
Quantitative and qualitative research
Social surveys and experiments are frequently viewed as prime examples of quantitative research.
Case studies are often seen as prime examples of qualitative research.
The purpose of research design is to reduce the ambiguity of much research evidence. Rather
than seeking evidence that is consistent with our theory we should seek evidence that provides a
compelling test of the theory. There are two related strategies for doing this: eliminating rival
explanations of the evidence and deliberately seeking evidence that could disprove the theory.
Plausible rival explanations
Good research design will anticipate competing explanations before collecting data so that relevant
information for evaluating the relative merits of these competing explanations is obtained. Too often
people do not even think of alternative hypotheses and simply conclude that since the evidence is
consistent with their theory then the theory is true.
This form is reasoning commits to the logical fallacy of affirming the consequent. This form of
reasoning has the following logical structure:
- If A is true then B should follow.
- We observe B.
- Therefore A is true.
Falsification: looking for evidence to disprove the theory
Rather than asking ‘what evidence would constitute support for the theory?’, ask ‘what evidence
would convince me that the theory is wrong?’. We need to frame our propositions and define our
terms in such a way that they are capable of being disproven.
2
,The provisional nature of support for theories
The more alternative explanations that have been eliminated and the more we have tried to
disprove our theory, the more confidence we will have in it, but we should avoid thinking that it is
proven. However, we can disprove a theory. The logic of this is:
- If theory A is true then B should follow.
- B does not follow.
- Therefore A is not true.
2: Tools for research design
Before design
Researches must be clear about their research question before developing a research design.
Focusing and clarifying the research question
The first question to ask is, ‘What question am I trying to answer?’.
Focusing descriptive research questions
To narrow the focus of descriptive research we need to specify the scope of what is to be
described. The following guidelines help narrow down a descriptive research topic into a
researchable question.
I. What is the scope of the core concepts?
II. What is the time frame for the description?
III. What is the geographical location for the description?
IV. How general is the description to be?
V. What aspect of the topic are you interested in?
VI. How abstract is your interest?
VII. What is the unit of analysis/the ‘thing’ about which we collect information and from which we
draw conclusions?
Focusing explanatory research questions
In framing explanatory questions we need to further specify our focus. The question must be clear
about the style of explanatory research and identify which causes or consequences it will
investigate.
- Searching for causes of effects:
This is the least focused type of explanatory research. It involved identifying the core
phenomenon and then searching for causes or consequences of this.
- Exploring a simple causal proposition:
A more focused research question will specify a particular causal proposition to evaluate.
- More complex causal models:
We might develop more complex models that spell out some of the mechanisms in the causal
chain.
When clarifying a research question it is helpful to draw diagrams. It is also helpful to ask four key
questions:
I. What am I trying to explain (i.e. what is the Y variable)?
II. What are the possible causes (what are the X variables)?
III. Which causes will I explore?
IV. What possible mechanisms connect the presumed causes to the presumed effects (what are
the intervening variables)?
3
,Another way of framing research questions is to formulate different ways of understanding a
phenomenon and then compare which of these competing approaches best fits the facts.
Idiographic and nomothetic explanations
Nomothetic explanations are partial explanations of a class of cases rather than a ‘full’
explanation of a particular case. They involve an examination of relatively few causal factors and a
larger number of cases and develop as complete an explanation of each case as possible. They
involve examining as many factors as possible that contribute to the case including unique factors.
The ideographic explanation would provide us with a good understanding of the case while the
nomothetic explanation would provide an understanding of the influence of a factor.
Identifying plausible rival theories
There are two main types of rival hypotheses and these suggest ways of anticipating alternative
explanations.
Theoretical and substantive rivals
There are several sources of alternative explanations:
- The theoretical literature
- Other researchers
- Practitioners, key informants, policy makers, advocates
- Own experience, hunches, and intuitions
Technical/methodological rivals
If findings are likely to be due to poor measurement then any theoretical interpretation will be
unconvincing. Good research minimises the threat from these sources. Types of rivals that will be
examined:
I. Demand characteristics of the situation.
II. Transient personal characteristics (respondent’s mood, attention span).
III. Situational factors (anonymity, gender, age).
IV. Sampling of items (are the concepts well measured?).
V. Nature of the sample (can we generalise from the sample?).
VI. Lack of clarity from the instrument (are the questions clear and unambiguous?).
VII. Format of data collection (are the questions clear and unambiguous?).
VIII.Variation in the administration of the instrument (in studies tracking over time).
IX. Processing/analysing errors.
Operationalisation
Concepts are, by their nature, not directly observable. To use concepts in research we need to
translate concepts into something observable, something we can measure. This involves defining
and clarifying abstract concepts and developing indicators of them.
Measurement error
Indicators must meet two fundamental criteria: they must both be valid and reliable. A valid
indicator in this context means that the indicator measures the concept we say it does. Reliability
means that the indicator consistently comes up with the same measurement.
The validity of a measure depends both on the use to which it is put and on the sample for which it
is used. There are three basic ways of assessing validity.
4
, - Criterion validity is best suited to situations where there are well-established measures of a
concept that need adapting, shortening or updating. It involves comparing the way people rate
on the new measure with how they rate on well-established measures of the concept. If ratings
on the new measure match those of an established measure we can be confident of its validity.
Criterion validity has two limitations:
- It requires that the established benchmark is valid.
- There are no established measures for many social science concepts.
- Content validity evaluates how well the measures tap the different aspects of the concept as
we have defined it. Given disagreement about the ‘content’ of many social science concepts it
can be difficult to develop measures that have agreed validity. Even if we can agree on the
concept and measure it using a whole battery of questions, we then face the problem of the
relative importance of the various components of the measure.
- Construct validity relies on seeing how well the results we obtain where using the measure fit
with theoretical expectations.
In the final analysis we will need to argue for the validity of our measures.
Forms of measurement error
Error can take different forms and the consequences of error will vary depending on its form.
These forms of error are random, constant and correlated.
- Random error is that which has no systematic form. It means that in some cases a
measurement for a variable might be too low while in others it is too high. Because random error
does not distort means and is uncorrelated with other factors, it is less serious than other forms
of errors.
- Constant error occurs where there is the same error for every case.
- Correlated error takes place when the amount and direction of errors vary systematically
according to other characteristics of respondents.
A crucial goal of the design and administration of survey instruments is the minimisation of the
various forms of measurement error.
3: Causation and the logic of research design
The main reason why it is difficult to establish causal relationships is because we cannot actually
observe one phenomenon producing change in another. The purpose of research design in
explanatory research is to improve the quality of our causal inferences.
Inferring causal relationships
Criteria for inferring cause
In order to infer that a probabilistic causal relationship exists between two variables, two basic
criteria must be met. First, there must be co-variation of causal and outcome variables. Second,
the assertion that one variable affects the other must make sense.
Co-variation
If X causes Y then people who differ from one another on X should tend to differ from one another
on Y. When two variables or events are correlated but not causally related the relationship between
the two variables is said to be spurious.
5
