Compiled by the Statistical Information and Research (SIR) Unit
March 2018
, PREFACE
The National Senior Certificate (NSC) examinations are set and moderated in
part using tools which specify the types of cognitive demand and the
content deemed appropriate for Geography at Grade 12 level. Until
recently, the level of cognitive demand made by a question was considered
to be the main determinant of the overall level of cognitive challenge of an
examination question.
However, during various examination evaluation projects conducted by
Umalusi from 2008-2012, evaluators found the need to develop more complex
tools to distinguish between questions which were categorised at the same
cognitive demand level, but which were not of comparable degrees of
difficulty. For many subjects, for each type of cognitive demand a three-level
degree of difficulty designation, easy, moderate and difficult was developed.
Evaluators first decided on the type of cognitive process required to answer a
particular examination question, and then decided on the degree of
difficulty, as an attribute of the type of cognitive demand, of that
examination question.
Whilst this practice offered wider options in terms of easy, moderate and
difficult levels of difficulty for each type of cognitive demand overcame some
limitations of a one-dimensional cognitive demand taxonomy, other
constraints emerged. Bloom’s Taxonomy of Educational Objectives (BTEO)
(Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956) and the Revised Bloom’s
Taxonomy are based on the assumption that a cumulative hierarchy exists
between the different categories of cognitive demand (Bloom et al., 1956;
Bloom, Hastings & Madaus, 1971). The practice of ‘levels of difficulty’ did not
necessarily correspond to a hierarchical model of increasing complexity of
cognitive demand. A key problem with using the level of difficulty as an
attribute of the type of cognitive demand of examination questions is that,
questions recognised at a higher level of cognitive demand are not
ii
,necessarily categorised as more difficult than other questions categorised at
lower levels of cognitive demand. For example, during analyses a basic
recognition or recall question could be considered more difficult than an
easy evaluation question.
Research further revealed that evaluators often struggled to agree on the
classification of questions at so many different levels. The finer categorization
for each level of cognitive demand and the process of trying to match
questions to pre-set definitions of levels of difficulty made the process of
making judgments about cognitive challenge overly procedural. The
complex two-dimensional multi-level model also made findings about the
cognitive challenge of an examination very difficult for Umalusi Assessment
Standards Committee (ASC) to interpret.
In an Umalusi Report, Developing a Framework for Assessing and Comparing
the Cognitive Challenge of Home Language Examinations (Umalusi, 2012), it
was recommended that the type and level of cognitive demand of a
question and the level of a question’s difficulty should be analysed
separately. Further, it was argued that the ability to assess cognitive
challenge lay in experts’ abilities to recognise subtle interactions and make
complicated connections that involved the use of multiple criteria
simultaneously. However, the tacit nature of such judgments can make it
difficult to generate a common understanding of what constitutes criteria for
evaluating the cognitive challenge of examination questions, despite
descriptions given in the policy documents of each subject.
The report also suggested that the Umalusi external moderators and
evaluators be provided with a framework for thinking about question difficulty
which would help them identify where the main sources of difficulty or ease in
questions might reside. Such a framework should provide a common
language for evaluators and moderators to discuss and justify decisions
about question difficulty. It should also be used for building the capacity of
novice or less experienced moderators and evaluators to exercise the
iii
, necessary expert judgments by making them more aware of key aspects to
consider in making such judgments.
The revised Umalusi examination moderation and evaluation instruments for
each subject draw on research and literature reviews, together with the
knowledge gained through the subject workshops. At these workshops, the
proposed revisions were discussed with different subject specialists to attain a
common understanding of the concepts, tools and framework used; and to
test whether the framework developed for thinking about question difficulty
‘works’ for different content subjects. Using the same framework to think
about question difficulty across subjects will allow for greater comparability of
standards across subjects and projects.
An important change that has been made to the revised examination
evaluation instrument is that the analysis of the type of cognitive demand of
a question and analysis of the level of difficulty of each question are now
treated as two separate judgments involving two different processes.
Accordingly, the revised examination evaluation instrument now includes
assessment of difficulty as well as cognitive demand.
iv
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through EFT, credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying this summary from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller ACTUALSTUDY. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy this summary for R142,63. You're not tied to anything after your purchase.
