Recruiting the Right Subject Matter Experts (SMEs)
Selecting the right subject matter experts (SMEs) is one of the most substantial decisions a credentialing program can make. SMEs shape everything from practice analyses and test content to standard setting and scoring decisions, making their selection foundational to exam validity, fairness, and defensibility.
As discussed during Quadterion’s Coffee & Credentials: SME Selection Secrets session, effective SME recruitment is not about finding the most available or well-known experts. It’s about intentional design, alignment with purpose, and disciplined documentation.
Recruiting Subject Matter Experts Starts with Purpose
A consistent message from the Quadterion team was that a SME panel’s purpose must drive its composition. Ian Hembry, principal psychometrician at Quadterion, emphasized that organizations should first clarify the mission of the credentialing program and the specific goal of the panel before identifying candidates. A practice analysis panel, for example, requires different expertise than an item-writing or standard-setting group.
When purpose is clearly defined up front, organizations are better positioned to identify which professional backgrounds, experience levels, and perspectives are relevant and avoid assembling panels that are misaligned with the work at hand.
Design Panels Intentionally and Representatively
Jim Henderson, former NCCA Chair and Quadterion Advisory Board member, reinforced that no single individual can represent an entire profession. Because practice evolves rapidly and varies across settings, panels, rather than individuals, are required to ensure accurate and current representation of the field.
Intentional panel design includes:
- Defining the demographic and professional characteristics that matter most for the credential
- Establishing target composition goals in advance
- Recruiting with those targets in mind rather than relying on convenience or familiarity
This approach aligns with accreditation expectations and strengthens the legal defensibility of credentialing decisions.
Avoid Common Subject Matter Expert Recruitment Biases
Several Quadterion panelists noted that bias in SME selection often enters quietly through overreliance on “people we know” or the repeated use of the same trusted volunteers. David Chen, senior psychometrician at Quadterion, pointed out that while familiar SMEs are often capable contributors, repeatedly drawing from the same pool can limit perspective and unintentionally exclude voices that reflect how practice is occurring across the profession.
Expanding recruitment strategies, such as leveraging professional networks, inviting recommendations, or rotating panel membership, can help organizations broaden perspectives while continuing to ensure expertise.
Use Academics Thoughtfully and Strategically
Educators and academics can be highly effective SMEs due to their subject knowledge, writing skills, and motivation to participate. However, as Dr. Henderson cautioned, their involvement in exam development requires careful boundaries. Exposure to live test content can create conflicts if not properly managed.
Mitigation strategies may include:
- Limiting the proportion of academics on exam development panels
- Restricting item exposure to specific domains
- Establishing confidentiality agreements and clear expectations around teaching practices
As Nathan Burns, exam development manager at Quadterion noted, conflicts of interest are not universal. They vary by panel purpose. Recognizing and addressing these differences is critical to maintaining exam integrity.
Document the Plan and the Reality of SME Recruitment
Across the discussion, documentation emerged as a cornerstone of effective SME recruitment. Dr. Hembry emphasized that organizations should document not only their recruitment plans, but also their outcomes, even when targets are not fully met.
Strong documentation includes:
- The purpose and charge of each panel
- Eligibility criteria and recruitment strategies
- Target composition goals and actual panel characteristics
- Rationale for deviations and adjustments
As our experts made clear, credentialing programs are not expected to achieve perfection, but they are expected to demonstrate thoughtful planning, good-faith effort, and transparency.
Looking for assistance with your SME recruitment strategy?