PAPER
WORDNET AND COSINE SIMILARITY BASED CLASSIFIER OF EXAM QUESTIONS USING BLOOM’S TAXONOMY
WordNet and Cosine Similarity based Classifier
of Exam Questions using Bloom’s Taxonomy
http://dx.doi.org/10.3991/ijet.v11i04.5654
K. Jayakodi1, M. Bandara2, I. Perera2, and D. Meedeniya2
1 Wayamba University, Kuliyapitiya, Sri Lanka
2 University of Moratuwa, Moratuwa, Sri Lanka
Abstract—Assessment usually plays an indispensable role in ble to evaluate the exam papers and limited knowledge of
the education and it is the prime indicator of student learn- the examiners about existing learning taxonomies and how
ing achievement. Exam questions are the main form of as- their exam questions fit into the taxonomies. Poorly de-
sessment used in learning. Setting appropriate exam ques- signed assessments usually fail to examine the achieve-
tions to achieve the desired outcome of the course is a chal- ment of course outcomes, which can lead to low quality
lenging work for the examiner. Therefore this research is graduates who do not fit with the employer expectations.
mainly focused to categorize the exam questions automati- Ultimately this can fail the goals of examination and result
cally into its learning levels using Bloom’s taxonomy. Natu- in degradation of the standards of degree program.
ral Language Processing (NLP) techniques such as tokeniza-
An exam question often falls into more than one level
tion, stop word removal, lemmatization and tagging were
of assessment categories of a given taxonomy. It is diffi-
used prior to generating the rule set to be used for this clas-
cult to categorize exam questions and even more difficult
sification. WordNet similarity algorithms with NLTK and
to identify the portion each taxonomy level of assessment
cosine similarity algorithm were developed to generate a
unique set of rules to identify the question category and the
belongs to. Therefore this research was carried out to
weight for each exam question according to Bloom’s taxon- generate an appropriate rule set using NLP and WordNet
omy. These derived rules make it easy to analyze the exam similarity algorithm, which was then combined with co-
questions. Evaluators can redesign their exam papers based sine similarity algorithm to assign the weight for each
on the outcome of this classification process. A sample of category of the question, according to Bloom’s taxonomy.
examination questions of the Department of Computing and The paper is arranged as follows: Section II presents the
Information Systems, Wayamba University, Sri Lanka was related literature on educational taxonomies and natural
used for the evaluation; weight assignment was done based language processing techniques used for exam question
on the total value generated from both WordNet algorithm evaluation. Section III elaborates the research methodolo-
and the cosine algorithm. Identified question categories gy and Section IV presents the results and analysis. Sec-
were confirmed by a domain expert. The generated rule set tions V, VI and VIII discuss research contributions and
indicated over 70% accuracy. conclude.
Index Terms—Question classification, Teaching and Sup- II. LITERATURE REVIEW
porting Learning, Bloom’s taxonomy, Learning Analytics,
Natural Language Processing, Cosine similarity A. Educational Taxonomy
Educational taxonomies can be used to measure the
I. INTRODUCTION achievement of course objectives. Taxonomy not only
does explain about the topics to be covered in a course but
Assessments are the systematic collection, review and
also help to understand the depth of each subject topic [5].
use of information in educational programs, undertaken
for the purpose of improving learning outcomes and stu- Once we identify the relationship of the chosen level of
dent development. Effective style of questions plays an the taxonomy and the course outcome we can assess stu-
dents at the chosen level through a suitable choice of ques-
important role in learner assessment. Through the art of
tions [6]. Educational researchers have developed several
thoughtful questioning teachers can extract not only factu-
taxonomies useful for the development of assessments,
al information, but also help learners in connecting con-
learning outcomes and educational resources. Out of those
cepts, making inferences, increasing awareness, encourag-
Bloom’s taxonomy [1] is in the foreground. In his study
ing creative and constructive thoughts. There are different
Bloom identified six main categories within cognitive
taxonomies that have been developed to identify the level
of the assessment being practiced such as Bloom’s [1] and domain. It starts from the lowest level (Fig. 1) and increas-
SOLO [2]; they are useful to identify the levels of the ingly moves to complex and abstract higher levels.
questions. While questions can be given throughout the Bloom’s categories were considered as the degree of diffi-
culty to achieve learning outcomes. The highest order is
course, mid semester and the end semester exam questions
classified as Evaluation and the lowest is classified as
often carry a considerable weight for the overall assess-
Knowledge level. It is expected that lowest level should
ment. When questions are prepared, there should be an
be mastered before moving into higher levels. Anderson et
effective balance between questions that assess the high
al [7] have already improved the noun list of Bloom’s
level of learning and questions that assess the basic level
taxonomy into a verb list (Table 1). Apart from that An-
of learning [3]. Often the exam questions used to assess
the level of the university students are at low cognitive derson identified the level of knowledge which makes
levels [4].This may happen due to the lack of tools availa- Bloom’s levels into a Matrix. For example, factual
142 http://www.i-jet.org
, PAPER
WORDNET AND COSINE SIMILARITY BASED CLASSIFIER OF EXAM QUESTIONS USING BLOOM’S TAXONOMY
knowledge, Conceptual knowledge, Procedural
knowledge and Metacognitive knowledge were identified
as the knowledge level dimensions [8].
Structure of observed learning outcome [SOLO] taxon-
omy [2] is another model, which concerns about student
understanding of the subject. SOLO provides a simple,
reliable and robust model for three levels of understanding
such as surface, deep and conceptual. It is up to the exam-
iner to define the type of content in the answer that is
expected. There are five main stages to be followed se-
quentially: Pre-structural, Uni-structural, Multi-structural,
Relational, and Extended Abstract. The lower level of
SOLO taxonomy is important to focus on individual items Figure 1. Bloom’s Taxonomy
of what is being assessed. The higher level is more con-
cerned with the broader range of elements or attributes to TABLE I.
ANDERSON’S REVISIONS ON BLOOM’S TAXONOMY
be examined.
Cognitive Verb list of Anderson Taxonomy
B. Educational taxonomy and NLP for exam evaluation Category
Description Verb list
Bloom’s taxonomy has been widely researched for stu- defines, describes, identifies,
dent assessments efficiency. Jerzy et al [9] analyzed the Recall or retrieve
knows, labels, lists, matches,
contents of laboratory exercises and lab tests to identify Categories previous learned
names, outlines, recalls, recognizes,
information.
the knowledge level. Bloom’s taxonomy of learning out- reproduces, selects, states
comes has been applied to classify the exam questions. In Comprehending comprehends, converts, defends,
general, disregarding the Bloom’s pyramid structure was the meaning, and distinguishes, estimates, explains,
found as the leading source of laboratory failure [9]. Turk- Understand interpretation of extends, generalizes, gives an
ish high-school physics examination and university en- instructions and example, infers, interprets, para-
trance examination questions were examined according to problems. phrases
Blooms’ taxonomy to identify the assessment levels of Use a concept in a applies, changes, computes, con-
those exam papers. It was revealed that university level new situation or structs, demonstrates, discovers,
Apply
unprompted use of manipulates, modifies, operates,
questions belong to higher levels whereas school ques- an abstraction predicts
tions belong to lower levels of the taxonomy [10]. Devel-
analyzes, breaks down, compares,
oping questions based on Bloom's hierarchy would be a Separates material
contrasts, diagrams, deconstructs,
productive way of ensuring the expected quality of student Analyze or concepts into
differentiates, discriminates, distin-
learning achievement. Higher skewness towards lower component parts
guishes, identifies, illustrates
levels of Bloom’s taxonomy can lessen the skill differen- appraises, compares, concludes,
tiation between a graduate and a first year undergraduate. Make judgments
contrasts, criticizes, critiques,
Thompson et al. [11] noticed that in case of science cours- Evaluate about the value of
defends, describes, discriminates,
ideas or materials
es there is a significant disagreement between academics evaluates, explains, interprets
in assigning questions into categories. For example, typi- categorizes, combines, compiles,
cal classroom tutorial problem ‘to calculate’ can fall into Builds a structure composes, creates, devises, designs,
understanding, application or synthesis categories depend- Create or pattern from explains, generates, modifies,
ing on the context. Therefore this research tries to provide diverse elements organizes, plans, rearranges, recon-
structs
an appropriate solution to assign the weights for each
question using revised Bloom’s taxonomy [7]. neural network techniques [16] have used NLP prepro-
Main purpose of NLP is to convert human language in cessing techniques.
to a formal representation that computers can understand. Chang [17] has extracted the verbs of a question to
NLP is used successfully in many fields such as infor- classify the question cognitive levels in which semantic
mation extraction, machine translation, text summariza- similarity was not taken into consideration. Question cate-
tion, search and human computer interfacing. These re- gorization with just keyword mapping is not the appropri-
search areas have used statistical NLP due to the easiness ate solution for every scenario. Auto marking [18], a tool
of interpretation [12]. Sentimental analysis is a field of developed for a LMS was capable of marking the student
NLP, which is used to identify and extract subjective answers submitted online. Student answers are often eval-
information from sources. NLP preprocessing techniques uated with the usage of semantic similarity algorithms
such as tokenization, stemming, tagging, lemmatization, available in WordNet.
chunking and parse generation were used in education
domain prior to applying semantic analysis techniques. 1) WordNet based algorithms for semantic similarity:
Learning Management System (LMS) support for users, Semantic similarity is a way to check the similarity be-
answering question and assessment generation, language tween documents, words and text by considering the dis-
learning and course preparation, subject evaluation, and tance between them. It is based on the likeliness of their
exam paper evaluation are few areas of education that meaning or semantic content as opposed to similarity,
NLP was used extensively [13]. Most of the question which can be estimated regarding their syntactical repre-
categorization techniques depend on the usage of NLP sentation. It consists of a number of algorithms, which is
preprocessing techniques. Question categorization meth- used to measure the semantic similarity and relatedness
odologies such as use of regular expression, term between a pair of concepts (synsets). There are two main
weighting [14], Support Vector Machine (SVM) [15], and ways to calculate the semantic similarity between two
ontologies: such as Edge-based and Node-based. Edge
iJET ‒ Volume 11, Issue 4, 2016 143
