Chapter 1 – Designing Research
A research question is a question that you have that you plan to answer, or at least
try to answer, by doing research.
Empirical research is any research that uses structured observations from the real
world to attempt to answer questions. So instead of trying to reason our way through what
drivers would do if given an additional highway lane, we try to observe the choices that
drivers take.
This book will focus on empirical research and, specifically, quantitative empirical
research. Quantitative empirical research is just empirical research that uses quantitative
measurements (numbers, usually). More data sets, fewer interviews. One particularly sticky
problem with quantitative empirical research is that the numbers that we observe often
don’t tell us exactly what we want to know. If the numbers we have don’t actually answer
the research question we have, what can we do? We have to carefully design the right kind
of analysis/research that will answer our question.
Research design is hard, and just because you want to answer a question doesn’t
mean there’s necessarily a straightforward way of doing it. But the worst that could happen
is that we’d figure out that the answer will be difficult to get.
Chapter 2 - Research Questions
A good research question is a question that can be answered, and for which having
that answer will improve your understanding of how the world works. What does it mean to
have a question that can be answered? It means that it’s possible for there to be some set of
evidence in the world that, if you found that evidence, your question would have a
believable answer. So for example, “What is the best James Bond movie?” can’t really be
answered. No matter what evidence you find, “best” is ambiguous enough that you can’t
even imagine the evidence that would settle the question for you. On the other hand,
“Which era of Bond movies had the highest ticket sales?” can definitely be answered. You
look at the ticket sales and see when they were highest. Evidence can tell you the answer to
this question. So now we have a question that can be answered. But does it improve our
understanding of how the world works? What this means is that the research question, once
answered, should tell you about something broader than itself. It should inform theory in
some way. Theory just means that there’s a why or a because lurking around somewhere. A
good test for whether a research question informs theory is to imagine that you find an
unexpected result, and then wonder whether it would make you change your understanding
of the world. So, does asking “Which era of Bond movies had the highest ticket sales?”
improve our understanding of how the world works? Maybe, for the right theory. Maybe we
have a theory that says that action movies were generally at their most popular in the 1980s.
1
, Let’s be honest, sometimes research questions also come from opportunity instead
of theory. For example, when you have a neat data set, think about what data is available to
you and whether any related research questions or theories come to mind. Or perhaps
you’ve learned about something unusual or interesting that has happened in the world and
then you might ask “what research questions would this allow me to answer?” and from
there you have a research question, and from there a theory.
A good research question takes us from theory to hypothesis, where a hypothesis is a
specific statement about what we will observe in the world. So, “if this is how the world
works, what would I expect to see in the world?”. An example could be: “People who wash
their hands will get sick less often.”
Okay so, you have a research question in mind. You know it can be answered with
data, and you’re pretty sure that if you get the answer to it, it will help you learn how the
world works. But is it really a good one? There are just a few things to check and consider:
Potential results. Consider the potential answers you might get and imagine what
kind of sense you’d make of that result, or what conclusion you would draw. If you can’t say
something interesting about your potential results, that probably means your research
question and your theory aren’t as closely linked as you think. Let’s say we do find that kids
who happen to play video games are more aggressive. Can we take that result and claim that
video games are a cause of aggression? Not really, as we just looked at which kids did play
video games, not whether video games are actually responsible for their aggression – maybe
kids who are more aggressive in the first place choose to play video games. So it could be
that the research question really isn’t linked to that theory very well.
Feasibility. A research question should be a question that can be answered using the
right data if the right data is available. But is the right data available? If answering your
research question is possible but requires too many difficulties (even though sometimes you
can get around it with a clever design), you might want to consider going back to the
drawing board.
Scale. What kind of resources and time can you dedicate to answering the research
question? Given a lifetime of effort and considerable resources, you might be able to tackle
massive questions like. Given the confines of, say, a term paper, you’re likely to do a much
more thorough job answering questions with a lot less complexity.
Design. An important part of evaluating whether you have a workable research
question is figuring out if there’s a reasonable research design you can use to answer it.
Keep it simple. A common mistake is to bundle a bunch of research questions into
one. “What are the determinants of social mobility?” There are many determinants of social
mobility. You’re unlikely to answer that question well. Instead, try “Is birth location a
determinant of social mobility?”
So, consider feasibility, scale, and design. Keep it simple, and think about whether
the results you might likely see would tell you anything interesting about the world. After all,
learning something interesting and new about the world is our goal.
2
, As there is data everywhere, why not skip the hard part of deriving a research
question from a theory and instead just see what sorts of patterns are in the data? This is
called “data mining.” You go to the data, look for patterns, and report back; just look at the
data, see what’s in there, and work backwards. Sounds good, right? Well, data mining turns
out to be very good at some things, but also very bad at others. The kinds of things that data
mining is good at are in finding patterns and in making predictions under stability. Seeing
patterns in data can give you ideas for research questions that you can examine further in
other data sources. Data mining is also probably the best angle to take when we don’t care
about why. The kinds of things that data mining is less good at are in improving our
understanding, or in other words helping improve theory. But why does data mining have
difficulty helping theory? There are a few main reasons. One of the reasons is that data
mining, by definition, focuses on what’s in the data, not why it’s in the data. Another reason
is that, because it’s so focused on the data, data mining doesn’t really deal in abstraction.
For example, data mining would be great at noticing that it sees a lot of flat bits on top of
straight-up-and-down bits, but it would not be good at developing “chair theory” for us.
False positives are another reason why data mining can be dangerous. Something is going to
pop up as related by random chance if you check enough stuff. That’s one major danger in
proceeding in your work without starting with a solid research question. Concluding, data
mining isn’t bad. It’s just bad as a final step if you’re trying to explain the world. It can still
work as a source of ideas.
Article by D.A. Whetten (1989)
The intent is not to create a new conceptualization of theory, but rather to propose
several simple concepts for discussing the theory-development process. My motivation is to
ease the communication problems regarding expectations and standards, which result from
the absence of a broadly accepted framework for discussing the merits of conceptual writing
in the organizational sciences.
This article is organized around three key questions: (a) What are the building blocks
of theory development? (b) What is a legitimate value-added contribution to theory
development? and (c) What factors are considered in judging conceptual papers?
So, what are the building blocks of theory development? A complete theory must
contain four essential elements:
What (describes). Which factors (variables, constructs, concepts) logically should be
considered as part of the explanation of the social or individual phenomena of interest? Two
criteria exist for judging the extent to which we have included the "right" factors:
comprehensiveness (are all relevant factors included?) and parsimony (should some factors
be deleted because they add little additional value to our understanding?). Sensitivity to the
competing virtues of parsimony and comprehensiveness is the hallmark of a good theorist.
3
, How (describes). Having identified a set of factors, the researcher's next question is,
How are they related? Operationally this involves using "arrows" to connect the "boxes."
Visual representation often clarifies the author's thinking and increases the reader's
comprehension. In particular, formal models aid theory developers and users to assess the
balance between parsimony and completeness. If all links have been empirically verified, the
model is ready for the classroom and is of little value in the laboratory. The mission of a
theory-development journal is to challenge and extend existing knowledge, not simply to
rewrite it.
Why (explains). What are the underlying psychological, economic, or social dynamics
that justify the selection of factors and the proposed causal relationships. During the theory-
development process, logic replaces data as the basis for evaluation. Theorists must
convince others that their propositions make sense if they hope to have an impact on the
practice of research. Combining the Hows and the Whats produces the typical model, from
which testable propositions can be derived. The primary difference between propositions
and hypotheses is that propositions involve concepts, whereas hypotheses require
measures. To avoid vacuous discussions, propositions should be well grounded in the Whys,
as well as the Hows and the Whats.
Who, Where, When. These conditions place limitations on the propositions
generated from a theoretical model. These temporal and contextual factors set the
boundaries of generalizability, and as such constitute the range of the theory. Unfortunately,
few theorists explicitly focus on the contextual limits of their propositions. In their efforts to
understand a social phenomenon, they tend to consider it only in familiar surroundings and
at one point in time. Theorists should be encouraged to think about whether their
theoretical effects vary over time, either because other time-dependent variables are
theoretically important or because the theoretical effect is unstable for some reason.
Sensitivity to context is especially important for theories based on experience. According to
the contextualist perspective (Gergen, 1982), meaning is derived from context. That is, we
understand what is going on by appreciating where and when it is happening. Observations
are embedded and must be understood within a context. Therefore, authors of inductively
generated theories have a particular responsibility for discussing limits of generalizability.
Then the second question: what is a legitimate value-added contribution to theory
development? Although, in principle, it is possible to make an important theoretical
contribution by simply adding or subtracting factors (Whats) from an existing model, the
additions or deletions typically proposed are not of sufficient magnitude to substantially
alter the core logic of the existing model. One way to demonstrate the value of a proposed
change in a list of factors is to identify how this change affects the accepted relationships
between the variables (Hows). Relationships, not lists, are the domain of theory. But, it’s the
Why’s that are probably the most fruitful, but also the most difficult avenue of theory
development. It commonly involves borrowing a perspective from other fields, which
encourages altering our metaphors and gestalts in ways that challenge the underlying
rationales supporting accepted theories. This profound challenge to our views of human
4
, nature, group development, organizational transactions, and so forth, generally precipitates
a broad reconceptualization of affected theories. This aspect of conceptual development is
particularly critical, and generally overlooked. Then the Who, When, and Where are left, as it
generally is insufficient to point out limitations in current conceptions of a theory's range of
application. Theorists need to understand the context to accommodate this new
information. The common element in advancing theory development by applying it in new
settings is the need for a theoretical feedback loop. Theorists need to learn something new
about the theory itself as a result of working with it under different conditions. Concluding,
three broad themes underlie this section:
Proposed improvements addressing only a single element of an existing theory are
seldom judged to be sufficient. Therefore, a general rule of thumb is that critiques should
focus on multiple elements of the theory. This approach adds the qualities of completeness
and thoroughness to theoretical work.
Theoretical critiques should marshal compelling evidence. This evidence can be
logical (e.g., the theory is not internally consistent), empirical (its predictions are
inconsistent with the data accumulated from several studies), or epistemological (its
assumptions are invalid – given information from another field).
In general, theoretical critiques should propose remedies or alternatives. Although
we can think of classic critiques in the history of science that stood on their own merits, the
typical debate in our field is less clear-cut. Consequently, critics should share responsibility
for crafting improved conceptualizations. Otherwise, it is difficult to know whether the
original is indeed inferior, or simply the best we can do in a very complex world.
Then the last question: what factors are considered in judging conceptual papers?
The following list of seven key questions, roughly in the order of frequency in which they are
invoked, summarizes the concerns raised most frequently by our reviewers. These questions
cover both the substantive issues discussed in the first two sections as well as several
formatting concerns. Together they constitute a summary answer to the broad question:
What's new? Does the paper make a significant, value-added contribution to current
thinking? Reviewers are not necessarily looking for totally new theories. However,
modifications or extensions of current theories should alter scholars' extant views in
important ways. In general, scope (how much of the field is impacted) is less
important in determining the merits of a contribution than is the degree (how
different is this from current thinking).
So what? Will the theory likely change the practice of organizational science in this
area? Are linkages to research evident (either explicitly laid out, or easily, reliably
deduced)? Does the paper go beyond making token statements about the value of
testing or using these ideas? Are solutions proposed for remedying alleged
deficiencies in current theories? The purpose of the standard theoretical paper
should be to alter research practice, not simply to tweak a conceptual model in ways
that are of little consequence.
5
