Automated Analysis of Learning Outcomes and Exam Questions Based on Bloom's Taxonomy

BERT Claude API Ollama zero-shot comparative study

• Systematic evaluation of NLP methods (Naive Bayes, LR, SVM, LSTM, BiGRU, BERT, RoBERTa, GPT-4, Claude, Gemini, Ollama) on a 600-sentence Bloom-labelled dataset.

• Key finding: model choice depends on data size; LLMs generalise well with no labelled data.

• Automatic classification of exam questions and learning outcomes

• 600 sentences labelled with six cognitive categories (Limited data according to study)

  • Knowledge
  • Comprehension
  • Application
  • Analysis
  • Synthesis
  • Evaluation

• Processing done with

  • Traditional ML models
    • Naive Bayes
    • Logistic Regression
    • Support Vector Machines
  • Recurrent Neural Network architectures
    • LSTM
    • BiLSTM
    • GRU
    • BiGRU
  • Transformer based models
    • BERT
    • RoBERTa
  • Large Language models
    • OpenAI
    • Gemini
    • Ollama
    • Anghropic

• Each model evaluated under different proprocessing and augmentation strategies

  • Synonym replacement
  • Word embeddings

• SVM - best performance - 94% accuracy, good recall and F1- scores. Minimal overfitting.

• OpenAI and Gemini 0.72 and 0.73 accuracy and commensurate F1-scores

• Methodology

  • Data Preprocessing
  • Data Augmentatoin

Relevance

• Directly validates the three-method comparison design (LLM, rule-based, TF-IDF). The paper tests Ollama and Anthropic models — the same tools you use — making it ideal for literature positioning.
• Cohen's kappa is used as the inter-rater metric here too.