Mission Critical: How the ABA Turns Exam Scores Into Certification Decisions

What happens from the time candidates leave exams to the moment they receive their results

ABA exams are momentous professional milestones on the path to board certification. Candidates spend years building their medical knowledge, practicing scenarios and training for assessments.

When a candidate walks out of a testing center, they might wonder, “What’s next? How long do I need to wait to learn about my results, and why does it take as long as it does?”

Board certification assesses candidates against a gold standard, and the ABA carefully vets all exam results to make deliberate scoring and certification decisions with that high caliber in mind. Computer-based, multiple-choice question exams include the In-Training Exams (ITE), BASIC and ADVANCED, which build on one another. The APPLIED Exam is the final step, in a live assessment of the clinical reasoning skills, judgment and aptitude required for board certification in anesthesiology.

ITE results are released six to eight weeks after the exam, while BASIC and ADVANCED take about four to six weeks. Candidates receive APPLIED results within about three weeks after the exam ends and see a pass or fail decision, as well as an explanatory report if they haven’t been certified.

What candidates don’t see is that from the time they finish their exam to the moment they receive their results, the ABA verifies them through a comprehensive process that includes psychometric analysis* of scores and quality control.

For all ABA examinations that result in a pass or fail decision, candidates are evaluated against an established performance standard, so that passing depends on demonstrating the knowledge and skills required for certification, not on out-performing a proportion of other examinees.

* Note: Psychometric analysis is broadly defined as the process of measuring and assessing psychological traits such as aptitude and cognitive abilities. In this instance, psychometric analysis refers to statistical evaluation and processing of exam scores with a focus on validity, reliability and defensibility.

Computer-Based Exams and APPLIED Exams Are Scored Differently

Computer-based and APPLIED exam scoring involve multiple layers of psychometric analysis and review before results are finalized. After candidates complete computer-based exams, ABA staff identify items that are not functioning as intended based on how each question performed across candidates after the exam. Board-certified anesthesiologist volunteers evaluate identified questions to determine the final set of questions that are scored using an established statistical model for multiple-choice examinations.

In contrast, APPLIED scoring focuses on evaluating and aligning examiner ratings across candidates using an established statistical model for performance examinations. APPLIED scores are processed on a shorter timeline, given where the exam falls in the certification process. It is the final step for many candidates, and results are released as quickly as possible while maintaining the same high standards of review that reflect the highest barometer for patient care quality.

APPLIED Scoring Begins During the Exam, Followed by Psychometric Analysis

The APPLIED Exam is scored by board-certified practicing anesthesiologists referred to as examiners. Examiners are trained to score candidates’ performance in the SOE and OSCE with a focus on rigor and high-quality clinical care. The exam consists of two components: the Standardized Oral Exam (SOE), where two examiners ask questions and score candidates’ performance, and the Objective Structured Clinical Exam (OSCE), where candidates navigate clinical scenarios and are scored by examiners watching remotely. Examiners complete SOE scoring after each session and complete remote OSCE scoring by the end of the exam week.

Immediately after the final candidate exits the exam, the ABA generates SOE and OSCE score reports and confirms the number of candidates assessed to produce a complete dataset that is then locked ahead of psychometric analysis.

The ABA scores the SOE and the OSCE using an existing statistical model called the many-facet Rasch model, which is suited to assessments where candidates are observed by different examiners across different stations. The model allows the ABA to statistically level the playing field, accounting for the fact that some stations are more challenging than others, and that there may be variability in examiner scoring. These factors are considered in the scoring process to ensure a candidate’s certification result is based only on their ability and knowledge rather than the stations and examiners they are assigned.

A System of Checks and Balances

Each candidate taking the SOE, designed to assess judgment, adaptability to unexpected clinical changes, organization and presentation of information, is scored by two examiners across various encounters. A candidate who takes the OSCE, which assesses communication, professionalism and technical skills related to patient care, is scored remotely by seven examiners, with one per station to gauge performance.

Multiple staff members verify critical elements at each stage, so no single interaction determines the outcome. When results fall close to the pass/fail line, the ABA revisits them for further review before moving forward. Examiner performance is also regularly reviewed to ensure they meet the standards for scoring consistency.

Two ABA staff psychometricians, a lead psychometrician and a quality assurance psychometrician (QA), work on a candidate’s score for each exam component, with four psychometricians working on each candidate’s exam. The lead and QA psychometricians each run a full analysis independently from one another using the many-facet Rasch model, and their results must match before they move forward.

Once scores are released, candidates who have not passed and wish to verify the accuracy of their scores may submit an assessment score verification request via ABA GO within six weeks of receiving their results.

All of these process elements come together as checks and balances, anticipating bias and human error and course-correcting before issues arise.

“The ABA’s APPLIED examination is meant to assess whether a candidate demonstrates the attributes of a consultant anesthesiologist (judgment, adaptability, application of knowledge and organization/presentation of thought) in managing clinical situations across the breadth and depth of our specialty,” said ABA Director and Assessments Committee Chair Mohammed Minhaj, M.D., MBA. “The rigor of the statistical analyses involved in our assessments process promote our core mission and ensure candidates meet the highest standards of our specialty.”

The same approach continues through individual case review to the release of results.

Reviewing Across Multiple Layers for Verification

In the days leading up to scores processing, the ABA verifies that each candidate meets eligibility, licensure, training pathway and policy requirements.

Multiple ABA staff members comb through the results, checking each candidate for policy adherence. They flag cases that necessitate follow-up and coordinate with credentialing and case management as needed, so each result aligns with the candidate’s record. This involves cross-team coordination as teams review records and resolve issues that would otherwise prevent candidates from being certified. In some cases, the ABA reaches out to candidates directly to follow up on documentation or notify them of requirements they must meet before they can be certified.

Finally, ABA technology staff oversee the final system upload of the results with quality control for accuracy. A full test run simulates the release of these results to make sure records update correctly, and any inconsistencies are identified to prevent error results.

These processes occur simultaneously as each team handles scoring, validation and upload, communicating with one another on a case-by-case basis.

For candidates who have worked diligently for years to reach exam day, the wait for results can feel significant. Behind the scenes, ABA staff deliberately considers and reviews every score to make certification decisions that are valid, reliable and defensible. For each candidate, the pursuit of board certification in anesthesiology represents an enormous undertaking. Every decision must measure up to the gold standard of practice reflected in that effort.

The Future State of Assessments

The ABA’s vision is to transform certification to shape the future of anesthesiology. To achieve that, the Board of Directors has identified seven strategies and more than 30 initiatives that make up our Strategic Plan. As we look to the future, the ABA is on its way to developing an innovative, digital-first assessment ecosystem to support training, initial and subspecialty certification and continuing certification. “Digital-first” refers to an approach where the entire assessment process, from development and delivery to scoring and feedback, leverages digital technologies.

###

About the American Board of Anesthesiology

The American Board of Anesthesiology (ABA) has served as the certifying body for anesthesiologists since 1938, partnering with physicians to elevate the specialty. Our mission is to advance the highest standards of the practice of anesthesiology. We administer initial and subspecialty certification exams and the MOCA continuing certification program, which facilitates lifelong learning with a focus on quality and safety.

The ABA’s work extends far beyond assessment. We prioritize innovation, research, technology and programming that promote exceptional patient care and clinical outcomes. Our dedicated staff collaborates with more than 700 board-certified anesthesiologist volunteers who develop exam and MOCA content, inform policy, serve as examiners and provide critical feedback. Their contributions across numerous committees and task forces drive the ABA’s strategic progress and help shape the future of the specialty.

Based in Raleigh, NC, we are a nonprofit organization and a Member Board of the American Board of Medical Specialties (ABMS).