1
Chapter 3: Data Collection Methods
There are numerous measurement methods that we can use to measure behavioural attributes in
the work context. No matter which method is used, all of have to be proven to be reliable and
valid.
If we adopt a broad definition of the term test, then we can conclude that testing includes all forms
of measurement (Murphy & Davidshofer, 2001). We can divide the measurement methods into four
groups, namely: direct observation, interviews, questionnaires and standardized psychological tests.
In Section 3 the focus is on direct observation, interviews and questionnaires. Standardized
psychological tests are discussed in Section 4.
3.1 Direct observation
Direct observation entails watching a person or persons while they are engaged in certain activities
and drawing conclusions about their behavior. The main problem with observation is that
observation can be very time consuming and labor intensive. It can also be unreliable.
In order to increase the reliability of observation more than one observer should be used and
observers should be well trained. Specific guidelines on what should be observed should also be
compiled. Examples of observation include: observation of on-the-job activities, role-play, the in-
basket exercise, and the leaderless group technique.
3.1.1 Observation of on-the-job activities
In this type of observation an observer watches an employee or employees for a period of time
while they are performing their daily activities. On-the-job observation can be valid if we specify
beforehand what will be observed. We should also plan when observations will take place and for
how long. The problem with this type of observation is that employees often behave differently
when they know that they are being watched.
In order to increase the reliability of on-the-job observation it is necessary for the observer to be
well trained and to remain as unobtrusive as possible. For example, the observer can stay out of the
way by sitting in a corner of the room while taking notes and not disturbing the person whom he or
she is observing.
3.1.2 Role-play
Sometimes it is not practical to observe employee behavior in a real life work situation. An artificial
situation is therefore created in which the person is observed. We can define roleplay as a
reproduction (imitation) of actual job behaviors in a controlled situation.
Role-play can be used, among other things, to assess interpersonal skills. In roleplay participants are
provided with relevant background information. They then “accept the duties and responsibilities of
their roles and functions, and do the best they can in the situation in which they find themselves"
(Jones, 1982, p. 113).
An example of a role-play is a situation in which one person plays the role of a supervisor and
another person the role of a subordinate who has to be disciplined. Observers assess the person’s
competency to conduct the disciplinary interview. In order to increase the reliability and validity of
this method, role-play should be as close to real-life situation as possible. According to Robbins
(1989), role-play has high validity. Hauenstien (2005) concluded that role-plays are valid predictors
of job success but that they take time and observers have to be well trained.
Copyright: C Pietersen, 2013
, 2
3.1.3 In-basket exercise
An in-basket exercise is an assessment method used for hiring, developing and promoting
employees. During the assessment, participants work through a typical in-tray. The tray includes a
number of mails, telephone messages, documents and memos. Participants have a limited period of
time to set priorities, delegate tasks, organize their work schedule accordingly and respond to mails
and phone messages. They sort through and deal with materials related to tasks, problems or
situations occurring on a routine basis in the position. They are expected to perform the tasks or
describe what they would do to handle the documents found in the in-basket. To be successful, they
must possess the knowledge, skills and abilities required to do the tasks typically performed on the
job.
An in-basket test may be used to assess competencies in the areas of communication, organization,
interpersonal relations, decision-making, leadership, and adjustment to change.
Trained assessors observe and evaluate the performance of the participants.
Research has shown mixed results regarding the reliability and validity of in-baskets. At most the
reliability of in-baskets is modest and their validity marginal (Joines, 2007).
3.1.4 Leaderless group technique (LGT)
Organizational effectiveness depends to a large extent on the effectiveness of work groups/teams.
Many decisions, problem-solving, policy-making and planning in organizations are done teams. The
LGT can be used to assess leadership potential, decision-making, problem-solving and conflict
management.
In the LGT groups of four to six participants are given a task to do, such as solving a problem or
making a decision. Trained observers assess the behavior of the participants. They identify
individuals who are task orientated and those who are more focused on supporting and interacting
with other participants in the group. They also identify those participants who demonstrate
leadership, dominance and the ability to persuade others to support their points of view, and those
who prefer to be followers. Research has indicated that LGTs have high inter-rater and test-retest
reliability (Bass, 1954).
3.2 Interviews
Interviewing is the most frequently used assessment method in the public and private sectors.
Interviewing is defined as a measure for gathering information by means of a discussion.
Interpersonal skills and motivation are probably best evaluated by the interview. It provides an
opportunity to obtain information about a person that is not readily available via more formal
psychological assessment methods. Interviews are used for selection purposes and also for
counselling (to manage poor performance and to change an employee’s behavior and thought
patterns) and for performance appraisal. It is therefore important to conduct good interviews that
produce valid and reliable results. Interviewers should receive training in different interview
techniques and should aware relevant legislation. For example, an assessment measure should
ensure no discriminatory questions are asked in the interview. Examples of illegal question are
questions about a person’s national origin, religion, criminal record or marital status.
An interview is typically done face-to-face but could also be conducted via telephone or
teleconferencing. Interview questions could be close-ended, open-ended or a combination of the
two types of questions could be used.
Copyright: C Pietersen, 2013
