Methods and techniques of evaluation research
Book 1: Evaluation. A systematic approach – Rossi, Lipsey and
Freeman (2004)
Chapter 1: an overview of program evaluation.
Social programs = an organizatonned, plannned, an usually ongoing nforr ednsignned ro amnliorarn a
social problnm or improvn social coneditonss
Programs arn ednsignned ro bnnnfr rhn human conediton
Examplns of social programs in rhn US arn communiry hnalrh cnnrnrs in low-incomns arnas,
scholarships for chiledrnn in edisaedvanragns schools aned edinrary high-prornin aned high-calorin fooed
supplnmnnr for prngnanr womnn in Cnnrral-Amnricas
All of rhnsn programs havn bnnn nvaluarned on rhn ourcomnss
Program nvaluaton is focusned on:
- Thn nnned for a program
- Hnlp improving social programs
- Sruedy aned appraisn social programs
- Aededrnss rhn way programs arn concnprualizned aned implnmnnrned
- Sruedy rhn ourcomns social programs achinvn (snrvicn ednlivnry)
- Sruedy rhn nfcinncy of social programs
Program nvaluaton rypically involvns assnssmnnr of onn or morn of fvn program edomains:
1s Thn nnned for a program
2s Thn ednsign of rhn program
3s Program implnmnnraton aned ednlivnry snrvicn
4s Program impacr or ourcomns
5s Program nfcinncy
A brinf hisrory of nvaluaton
Bnforn Worled War II: rhnrn was a focus on social programs in neducaton aned public hnalrh, such as
initatvns ro rneducn morraliry aned morbiediry from infnctous edisnasnss Social scinntsrs wnrn using
rigorous rnsnarch mnrhoeds ro assnss social programs in a varinry of arnass
Afnr Worled War II: rhn focus was on numnrous major fnednral aned privarnly funedned programs ro
proviedn urban ednvnlopmnnr aned housing, rnchnological aned culrural neducaton, occupatonal rraining
aned prnvnntvn hnalrh actvitnss
- 1950 – 1960: program nvaluaton was morn aned morn nxpaneding knowlnedgn of rhn rnsnarch
mnrhoeds aned aedvancned sratstcal procnedurnss Thnrn was a massivn grow of artclns aned
books abour nvaluaton rnsnarchs
- 1970 – 1980: nvaluaton rnsnarch nmnrgned as a edistncr spncialry fnled in rhn social scinncnss
- 1980 – now: largn curbacks rargnrned social programss Evaluaton has bnnn in rhn miededln of a
conrnntous ednbarn abour rhn vnry concnpr of social inrnrvnntons Thnrn has bnnn morn
atnnton bnnn edrawn ro rhn balancn of bnnnfrs wirh cosrs of social programss Borh
edissatsfacton wirh nxistng programs aned shifs in politcal currnnrs will rnsulr in nnw aned
moedifned programs rhar comn forwared wirh promisn of bning morn nfnctvn aned lnss cosrlys
Ovnr tmn, nvaluaton rnsnarch haed a qualitative changes In irs narly ynars nvaluaton was shapned
mainly by rhn inrnrnsrs of social rnsnarchnrss In larnr sragns, hownvnr, rhn consumnrs (such as
1
,policymaknrs, program plannnrs aned aedminisrrarors) of nvaluaton rnsnarch nxnrcisned a signifcanr
influnncn on rhn fnleds Also, rhn importance of evaluation is changned ovnr tmns owaedays, many
fnednral, srarn aned local agnncins rngularly conrracr for program nvaluaton wirh univnrsiry
rnsnarchnrs or rnsnarch frms aned consulranrss Alrhough nvaluaton rnsnarch contnuns ro havn an
acaednmic siedn orinnrned rowared rraining, mnrhoedology, rhnory aned sruedy of rhn narurn aned nfncrs of
social programs, ir is now gnnnrally practcned in a conrnxr of policy making, program managnmnnr or
clinnr aedvocacys
Thn ednfning characrnristcs of program nvaluaton
Program evaluation = rhn usn of social rnsnarch mnrhoeds ro sysrnmically invnstgarn rhn
nfnctvnnnss of social inrnrvnnton programs in ways rhar arn aedaprned ro rhnir politcal aned
organizatonal nnvironmnnrs aned arn ednsignned ro inform social acton ro improvn social coneditonss
Program nvaluaton is edual:
- Dnlivnr a ednscripton of rhn pnrformancn of rhn program bning nvaluarned
- Somn sranedareds or crirnria for juedging rhn pnrformancn of rhn program
Cnnrral aim: ro consrrucr a valied ednscripton of program pnrformancn in a form rhar pnrmirs incisivn
comparison wirh rhn applicabln crirnrias
To fulfll rhis aim, rnsnarchns usn social rnsnarch mnrhoeds aned rhn accompanying sranedareds of
mnrhoedical qualirys Hownvnr, rhny also havn ro bn innovatvn aned improvisn as rhny atnmpr ro fned
ways ro garhnr nviednncn abour rhn social programss
Thn challnngns ro rhn nvaluaror arn ro march rhn rnsnarch procnedurns ro rhn nvaluaton
qunstons aned circumsrancns as wnll as possibln aned, wharnvnr procnedurns arn usned, ro apply
rhnm ar rhn highnsr possibln sraned fnasibln ro rhosn qunstons aned circumsrancnss
- Aedaptng nvaluaton ro rhn politcal aned organizatonal conrnxr
Thn nvaluaton plan is usually organizned arouned rhn qunstons posned abour rhn program by:
1s Thn nvaluaton sponsor = rhn pnrson, group or organizaton rhar rnqunsrs or rnquirns rhn
nvaluaton aned proviedns rhn rnsourcns ro coneducr irs
2s Sraknholednrs = inedivieduals, groups or organizatons rhar havn a signifcanr inrnrnsr in how
wnll a program functonss
Mosr ofnn rhn nvaluaror musr nngotarn wirh rhn nvaluaton sponsors aned sraknholednrs ro ednvnlop
aned rnfnn rhn qunstonss
In ednsigning rhn nvaluaton, rhnrn arn numbnr of organizatonal facrors rhar musr by raking inro
accounr:
- Thn availabiliry of aedminisrratvn coopnraton aned supporr
- Thn ways in which program flns aned edara arn knpr aned accnss pnrmitned ro rhnm
- Thn characrnr of rhn snrvicns proviedned
- Thn narurn, frnqunncy, eduraton aned locaton of rhn conracr bnrwnnn rhn program aned irs
clinnrs
Evaluaton informs social acton, conrributng informaton for planning aned policy purposns,
inedicatng whnrhnr cnrrain innovatvn approachns ro communiry problnms arn worrh pursuing or
ednmonsrratng rhn utliry of somn principln of profnssional practcns
Funedamnnral ro nvaluaton is rhn purposn ro inform actonn
Evaluaton rnsnarch in practcn
In practcn, applying rhn nvaluaton rnsnarch is an inhnrnnr conflicr bnrwnnn the requirements of the
systematic inquiry aned edara collncton associarned wirh nvaluaton rnsnarch vss the organizational
2
,imperatives of a social program ednvorned ro ednlivnring snrvicns aned mainraining nssnntal routnn
actvitnss
ngotarn a miededln way bnrwnnn optmizing rhn siruaton for rnsnarch purposns aned
minimizing rhn edisrupton causned ro normal opnratonss
Somn edilnmmas in rhn fnled ovnr rhn nvaluaton rnsnarch arn:
- Thn availabiliry of rnsourcns for rhn nvaluaton, rhn tmn linns for proeducing rnsulrs aned rhn
rnlatonships wirh rhn program aedminisrrarors aned nvaluaton sponsorss Social programs arn
nor rnsnarch laborarorins, hownvnr, aned nvaluarors musr nxpncr ro bn bufnrned abour by
forcns aned nvnnrs oursiedn rhnir conrrols Also, program circumsrancns aned actvitns may
changn eduring rhn coursn of an nvaluatons
- Scinntfc vss pragmatc nvaluaton: on rhn onn haned rhnrn is rhn vinw rhar high sranedareds of
scinntfc rnsnarch shouled bn mnr in rhn nvaluaton rnsnarch (vinw of Campbnll) vss rhn vinw
of rhar nvaluaton shouled bn ednedicarned ro snrving rhn informaton nnneds of program edncision
maknrs (vinw of Cronbach)s
- Thnrn is a wiedn edivnrsiry of pnrspnctvns aned approachns in rhn nvaluaton fnled rhar proviedn
litln frm guiedancn abour how ro procnned wirh an nvaluatons
Utlizaton of nvaluaton = rhn usn of concnprs aned fnedings of an nvaluaton by edncision maknrs aned
orhnr sraknholednrs whnrhnr ar rhn eday-ro-eday managnmnnr lnvnl or ar broaednr funeding or policy
lnvnlss
Thnrn arn rhrnn edifnrnnr pnrspnctvns on rhn utlizaton of nvaluaton:
- Pnrspnctvn 1 = Aedvocarn nvaluaton is ednsignned arouned rhn informaton nnneds of spncifc
sraknholednrs
- Pnrspnctvn 2 = Aedvocarn nvaluaton shouled conrriburn ro cumulatvn knowlnedgn abour social
inrnrvnntons
- Pnrspnctvn 3 = Aedvocarn nvaluaton shouled bn a combinaton of rhn rwo pnrspnctvns
ednscribned abovns
Chapter 2: Tailoring evaluations.
Onn of rhn mosr challnnging aspncrs of nvaluaton is rhar rhnrn is no ‘onn sizn frs all’ approachs Thn
nvaluaton ednsign musr, rhnrnforn, involvn an inrnrplay bnrwnnn rhn narurn of rhn nvaluaton
siruaton aned rhn nvaluaror’s rnpnrroirns
Any nvaluaton ednsign/plan is srrucrurned arouned 3 issuns:
- Thn qunstons rhn nvaluaton is ro answnr – concnrns such as rhn nnneds of rhn rargnrs ro
which a program is edirncrned aned whnrhnr rhn program is having rhn ednsirned impacr, bur also
concnrns/qunstons abour rhn cosrs aned nfcinncys
Target = rhn unir ro which a program inrnrvnnton is edirncrneds All such unirs wirhin rhn arna snrvned
by a program comprisn irs rargnr populatons Examplns arn pnrsons, familins or social groupss
Catchment area = rhn gnographical arn bning snrvned by rhn programs
- Thn mnrhoeds aned procnedurns rhn nvaluaton will usn ro answnr rhn qunstons – raking rhn
tmn, usnfulnnss aned crnedibiliry of rhn rools in accounrs
- Thn narurn of rhn nvaluaror-sraknholednr rnlatonship – rhnrn nnneds ro bn a plan for working
wirh sraknholednrs ro iednntfy aned clarify rhn issuns, coneducr rhn nvaluaton aned makn
nfnctvn usn of rhn fnedingss This inrnracton can bn nirhnr collaboratvn (1) wirh a sharned
rnsponsibiliry for borh partns or aedvisory (2), in rhn snnsn rhar rhn nvaluaror has rhn
rnsponsibiliry bur snnks nssnntal guiedancn aned informaton from sraknholednrss
Thnrn arn 3 fnarurns of rhn siruaton rhar rhn nvaluaton plan shouled rakn inro accounr:
3
, 1. The purpose of the evaluation
Thn prospnctvn nvaluaror musr atnmpr ro ednrnrminn who wanrs rhn nvaluaton, whar rhny wanr
aned why rhny wanr irs Thnrn can bn snvnral rnasons for pnopln ro initarn an nvaluaton, such as:
- Program improvnmnnr: rhis is also callned formative evaluation, bncausn of irs purposn ro
hnlp form or shapn rhn program ro pnrform bntnrs Thn auedinncn for formatvn nvaluatons
arn mosrly program plannnrs, aedminisrrarors, ovnrsighr boareds or funednrss Thn nvaluaror will
work closnly rognrhnr wirh program managnmnnr aned orhnr sraknholednrs aned rhnir
communicaton is rnlatvnly informals
Formative evaluation = nvaluatvn actvitns unednrraknn ro furnish informaton rhar will guiedn
program improvnmnnrs
- Accounrabiliry: rhis is also callned summative evaluation, which mnans rhar rhis is an
nvaluaton coneducrned ro ednrnrminn whnrhnr rhn nxpncratons arn mnr aned chnck rhn
program’s pnrformancns Thn fnedings of rhnsn nvaluaton arn inrnnedned for edncisionmaknrs
aned will influnncn signifcanr edncisions (such as contnuaton or allocaton of rhn program)s
Thn nvaluaror works mosrly alonn aned inednpnnednnrly aned rhn sraknholednrs will proviedn inpur,
bur arn nor partcipatng in rhn edirncrly edncision making eduring rhn nvaluatons arnr on rhn
nvaluaror sums up his fnedings in a writnn rnporr for rhn auedinncns Also, rhn communicaton
is rnlatvnly formals
Summative evaluation = nvaluatvn actvitns unednrraknn ro rnnednr a summary juedgnmnnr on cnrrain
critcal aspncrs of rhn program’s pnrformancn, for insrancn ro ednrnrminn if spncifc goals aned
objnctvns wnrn mnrs
- Knowlnedgn gnnnraton: rhis kined of nvaluaton is inrnnedned ro makn conrributons ro rhn social
scinncn knowlnedgn basn or bn a basn for signifcanr program innovatons Usually rhn mosr
rigorous mnrhoeds arn usned aned rhn fnedings (writnn edown in scholarly journals of confnrnncn
papnrs) arn inrnnedned for scholars, policymaknrs aned sponsors of rhn rnsnarchs
- Hiedednn agnnedas: rhis kined of nvaluaton can bn abour imprnssing funednrs or politcal
edncisionmaknrs aned wantng gooed public rnlatonss Thn nvaluaton musr nirhnr bn guiedned by
rhn politcal or public rnlatons purposns, which will liknly compromisn irs inrngriry, or focus
on program pnrformancn issuns rhar arn of no rnal inrnrnsr ro rhosn commissioning rhn
nvaluaton aned may nvnn bn rhrnarnning ro rhnms Thn nvaluaror couled choosn ro ednclinn or
rakn a consulranr rolns
2. The program’s structure and circumstances
o rwo programs arn iednntcal in rhnir organizatonal srrucrurn aned nnvironmnnral, social aned
politcal circumsrancnss Difnrnncns bnrwnnn rhn programs can occur in:
- Sragn of program ednvnlopmnnr: rhis is edifnrnnr for nnw aned alrnaedy nsrablishned programss
nw programs arn ofnn nvaluarned on rhnir social nnneds, program’s ednsign, ednfniton of rhn
rargnr populaton aned rhn nxpncrned ourcomnss Alrnaedy nsrablishned program’s nvaluaton arn
focusned on rhn nxrnnr ro which rhn objnctvns arn mnr, whnrhnr rhn programs arn wnll
implnmnnrned aned conform ro plan aned if rhny acrually rnach all of rhnir rargnr populatons
- Thn aedminisrratvn aned politcal conrnxr of rhn program: mosr nvaluarors arn nor frnn ro
nsrablish rhnir own ednfnitons of whar rhn program is abour/goals/objnctvns, bur havn ro
work wirh rhn nvaluaton sponsor, program managnmnnr aned orhnr sraknholednrs ro ednvnlop
rhnsn knowlnedgns Thnrn can bn conflicr bnrwnnn rhn pnrspnctvns of snvnral sraknholednrs
nvaluaror musr choosn onn pnrspnctvn ro work from (lnr’s say onn of rhn program funednrs)
or musr clarify rhn edifnrnncns bnrwnnn rhn sraknholednrs aned comn up wirh an own
pnrspnctvn (combining snvnral pnrspnctvns)s
4
