The SME’s Role in Item Development
This article explores what Subject Matter Experts (SMEs) do during the item development phase and why their involvement is foundational to exam quality.
In credentialing, an “item” is a test question. The item development phase is the most demanding, time-intensive phase of the exam development process. SMEs attend a training or kickoff where they receive instructions, examples, and assignments from an assessment developer or staff member on best practices and tips for item writing. They then rely on their experience from practice and their knowledge of concepts to construct scenarios, dilemmas, and other situations where candidates will be asked to demonstrate their competency.
SMEs may also be asked to review items written by other SMEs, revise their own items after feedback from other SMEs/mentors/editorial staff, and sit on review calls to reconcile any lingering issues. This is why item writing isn’t a task you hand off to generalists. The people writing and reviewing exam content must understand the profession from the inside out.
Three Core Contributions SMEs Make in Item Development
1. Writing Items Aligned to Competencies
SME-authored items begin from a place no psychometrician can manufacture: actual professional experience. Before the first question is written, a job task analysis (JTA), (sometimes called a practice analysis or role delineation study), identifies the competencies that matter most in the field. This becomes the exam blueprint: a map specifying which content domains the exam covers and how much weight each receives.
SMEs then translate those competencies into individual test items. They write questions grounded in real-world scenarios, realistic distractors (wrong answer choices), and the clinical or technical judgment a credentialed professional would actually need to exercise. An SME doesn’t just know what the right answer is; they know what a plausible but incorrect answer looks like, and why a less than minimally competent candidate might choose it.
This authenticity is the core value SMEs provide. The scenarios are true to practice, the stakes embedded in the questions feel real, and the knowledge being tested reflect situations where candidates must demonstrate their ability to recognize and demonstrate safe, competent performance.
Item writing workshops typically open with training: what makes a strong item, how to write clear and unambiguous stems, how to construct distractors that reflect realistic misconceptions rather than trick questions, and how to avoid technical pitfalls like absolute language or cue words that inadvertently reveal the correct answer. After that training, SMEs author questions individually or in small groups, with higher-stakes programs typically requiring each item to be independently reviewed by at least one additional SME.
2. Reviewing Items for Relevance, Clarity, and Bias
Writing items is only the first step. Before any question can enter the active item bank, it undergoes a multi-layered review, with SMEs central to that review at every stage.
During content review, SMEs evaluate items against three key criteria:
Relevance means the item directly assesses the knowledge or skill it targets, without interference from extraneous factors. An item that technically tests the right domain but requires unusual cultural knowledge or tests vocabulary more than content lacks relevance to the credential, and SMEs are positioned to spot that drift.
Clarity means the question is unambiguous. SMEs read items through the lens of a working professional, which surfaces confusing phrasing, undefined jargon, or scenarios that would be interpreted differently depending on a candidate’s practice setting or geographic region. An item that seems clear to a psychometrician reviewing it in isolation may be read entirely differently by someone who’s actually encountered that clinical situation.
Bias is among the most important things SME review panels can catch. Items that disadvantage candidates based on gender, ethnicity, cultural background, or practice geography compromise the exam’s validity. A diverse SME panel is the most effective tool for surfacing items that, however unintentionally, reflect the blind spots of any single author’s perspective. This is why representative panel composition (drawing SMEs from varied demographics, practice settings, and regions), is more than an equity consideration. It’s a technical quality requirement.
3. Calibrating Content to Practice
One of the most critical but frequently mismanaged aspects of credentialing exam development is getting the difficulty level right. An exam that tests advanced expert knowledge when it should be measuring entry-level competency doesn’t protect the public. It inflates failure rates, disadvantages candidates through unnecessary barriers, and can expose the credentialing organization to legal challenges.
SMEs are uniquely positioned to calibrate this. They can distinguish between what a new practitioner needs to do safely on day one versus what a seasoned expert has learned over a decade of specialized work. This calibration centers on the concept of the minimally qualified candidate (MQC)—the person who knows just enough to earn this credential. Items should be pitched at a difficulty level where that candidate, and any higher scoring candidate, would be expected to answer correctly, while those below that threshold would not. The decision of where to draw the line between qualified and not qualified is the cut score.
Getting the right SMEs in the room is essential to calibrating content accurately.
The Division of Labor: SMEs and Psychometricians
A credentialing exam is not produced by SMEs alone, nor by psychometricians alone. It’s the product of a structured collaboration between two kinds of expertise that don’t overlap much but depend on each other completely.
SMEs bring professional realism. They know what the job requires, what new practitioners get wrong, which scenarios pose genuine safety risks, and which knowledge is truly foundational versus merely interesting.
Psychometricians bring technical qualities. They ensure items meet best practice by checking for clear language, the structure and length of the stem and response options, and any unnecessary cueing that would give away the answer to candidates.
Neither can replace the other. An SME who writes a beautiful, accurate, clinically realistic item has nonetheless produced a question that could still perform poorly. A psychometrician reviewing that same item for statistical alignment cannot tell you whether the scenario reflects actual practice. It takes both.
Why SME Item Development Is Worth Getting Right
When items accurately reflect the appropriate level of competency, are free from bias, and are grounded in real-world professional judgment, a credentialing program can be confident that the content of the exam is representative of the profession. When the process is rushed, or conducted under less than best practice, the exam may not be a reliable indicator of candidate competency. It becomes a less defensible one, a less fair one, and ultimately a less meaningful one.
SME involvement in item writing and test development is not an administrative checkbox. It’s where the credentialing program either builds or weakens its validity evidence.
Need support in your item development process? We’ve got you.