Calibration of Rating Score Workshop (SoM Faculty Members)
Date: 30 August 2022 ( Tuesday)
Time: 2:00 pm – 6:00 pm
Training Platform: Zoom
Facilitators:-
- Professor Dr Shahid Hassan
- Dr Malanashita Ganeson
Categories:
- Assessment (A)
Roles:
- Role 4: Assessor
- Role 8: Professional & enquirer into own competence
Workshop Objective:
- The main purpose of this workshop is to create awareness about issues associated with subjective marking of assignments as extended response question (ERQ/MEQ) and restricted response question (RRQ/SEQ) in UG/PG programs.
- The other purpose is to produce consistent performance of the same or two different students doing equally well in their Essay test, receive the same or similar scores from evaluators, no matter where and when they take the examination.
- Yet another purpose is to achieve the similar learning outcome of assessment as the evidence of students’ performance, irrespective of test items marked by single or double markers.
Workshop Overview:-
Calibration is a process in which measurement instrument or equipment’s accuracy is compared with a known or proven standard. In calibration process of test items, teachers are trained to score students work using a common scoring rubric/model answer in order to ensure consistency and accuracy in scoring.
In an outcome based education (OBE) program it is important to have measurement accuracy and reliability of assessment to determine the outcome of students’ performance evidenced by constructive alignment. The ability to accurately and consistently assess students’ performance is imperative in OBE. However, in view of ever growing undergraduate and other postgraduate program in IMU in terms of future intake, marking of test items based on essays will be challenged by problem of accuracy and consistency of measurement marked by multiple assessors.
The assessment should be well aligned with students’ demonstrating the specific competencies and the task requirement of the curriculum, which is critical to a program. This becomes even more critical to program when performance is measured by single measurement tool, which is subjective like extended response question (ERQ) /assignments, limited items number (MEQ) and multiple learning outcome addressed by one ERQ. This creates the need for calibration training of evaluators to score students assessment with accuracy and consistency.
The calibration should be an ongoing training program that gives an opportunity to evaluators to review and rate the assessment as assignments/ERQ (MEQ) or RRQ (SEQ). Calibration should be developed as program in an organized and systemic manner to maintain standards on reliability of examination questions with acceptable inter-ratter agreement and insignificant discrepancy among the evaluators.
Workshop Learning Outcomes:
- Describe the purpose and importance of calibration in rating scores (C1, C2).
- Demonstrate a shared level of effective rating of essay items as ERQ or RRQ (C3).
- Analyse own performance as ratter for marking students’ assignment with evidently reliable score (C4).
- Evaluate the rating score based on DAG system in providing feedback on rating judgment (C5).