The scoring of trainee performance in stations
In each station, an assessor will score the trainee’s response to the scenario using a pre-defined set of criteria for the particular station. Assessors will also provide a global rating for the trainee, which is their professional judgement of the overall performance of the trainee in response to the scenario. Each station will have its own particular pass mark. The process for determining station pass marks is outlined further below.
Determining an overall outcome for the assessment
The mean of all of the station pass marks within a specialty will determine the overall pass mark for the assessment for each specialty. Trainees scoring more than the overall pass mark will be awarded a pass for the assessment (provided that they have also passed at least 3 of the specialist stations and 1 of the core stations). Trainees therefore may not need to pass every station to achieve the minimum threshold required to be successful in the assessment.
Trainees who receive a fail outcome in the assessment will have an opportunity to re-sit, according to the terms outlined in the Assessment Regulations.
How are station pass marks determined?
The pass marks for individual stations will be determined by a method called the Borderline Regression Methodology (BRM). This is a standard setting method widely used in exams for trainees in many healthcare professions. BRM uses expert assessor judgments of ‘borderline’ candidates, integrated with a statistical analysis of station criteria scores, to determine a station’s pass mark. BRM is considered to be examinee- or trainee-centred because it derives pass marks from actual trainee performance across stations with different degrees of difficulty.
Some key features and benefits of using the Borderline Regression Methodology are:
- It is evidence-based, reflecting actual performance
BRM uses actual trainee performance data and assessor judgments to derive pass marks.
- It generates station-specific pass marks
Since the pass mark for each station is based upon actual trainee performance data and assessor judgements, each station will have its own specific pass mark, reflecting differences in difficulty.
- It combines quantitative and qualitative data
BRM integrates station-specific criteria or checklist scores (quantitative) with overall assessor global rating judgements (qualitative), to provide a more holistic view of trainee performance.
- It is influenced by expert assessor judgments of minimal competence
Since BRM anchors pass marks around the performance of ‘borderline’ candidates, it aligns pass and fail outcomes with assessors’ understanding of minimal competence.
- It can reduce subjectivity and bias
By basing pass marks on a statistical analysis that focuses on borderline cases, BRM minimizes the influence of extreme judgments and derives a pass mark from actual performance trends across a cohort rather than a hypothetical standard that is set independently of the assessment event.
Qpercom, the company who provide the assessment platform within which the Final Year Assessment was conducted, have published a brief summary of the methodologies involved in BRM with accompanying academic references.
In cases where the School decides that the application of BRM cannot produce a robust and defensible set of outcomes for a given specialty (for example, where the sample size is too small) then the preferred approach will be to establish an overall pass mark for the specialty which corresponds to the historic average failure percentage. Additionally in these cases, Lead Assessors will be invited to discuss the outcomes with the School in order to confirm and refine the acceptability of this approach as it applies to individual candidates.