When to Retire Exam Items
Have you ever reviewed the statistics for an exam question and realized it’s not performing as intended? Your carefully crafted item is problematic, even though it took weeks to develop and went through multiple expert reviews. Now you need to decide what to do. When should you retire the exam item?
In exam development, retaining underperforming items is inefficient and risky. Understanding when and how to retire items is essential to maintaining a defensible credentialing program.
Warning Signs: When Exam Items Should Be Retired
These specific indicators should prompt immediate consideration for item retirement:
Consistently Poor Statistics
When an item shows low point-biserial correlation, negative discrimination, or extreme difficulty across multiple administrations, that’s clear evidence of a problem. While one poor performance might be a statistical anomaly, patterns indicate fundamental flaws. If candidates who demonstrate overall competence are getting the question wrong while weaker candidates are getting it right, the item is actively undermining your exam’s validity.
Outdated Content
As professional practices evolve, regulations change, and technology advances, it’s essential to keep exam questions up to date and relevant. Items testing obsolete procedures, laws, professional standards, or materials (tools, medications, etc.) must be removed regardless of their statistical performance.
Successful Challenges or Bias Flags
If a candidate has successfully appealed an item, or if your psychometrician has identified potential bias through differential item functioning (DIF) analysis or other methods, continuing to use that item creates unnecessary legal exposure. Validity concerns, once substantiated, cannot be ignored.
Insufficient Documentation
Every item on a credentialing exam must have a clear link to job task analysis findings and blueprint specifications. If you cannot trace an item back to documented practice requirements, then it lacks the foundation needed for defensibility. This problem is common with inherited item banks.
Common Barriers to Exam Item Retirement
Understanding why item retirement gets delayed can help you better address these barriers:
Sunk Cost Fallacy
The sunk cost fallacy leads us to continue with something that’s not working because we’ve devoted time and resources to the project. Focus on future impact, not past investment. The time and resources invested in developing an item are already spent, and continuing to use a flawed item doesn’t recover that investment. It compounds the problem by putting the validity of your entire exam at risk.
Item Bank Capacity Concerns
A smaller bank of high-quality items is always preferable to a larger bank containing problematic questions.
In high-stakes testing, quality cannot be sacrificed to meet quantity targets. If your bank is lacking, prioritize developing new items rather than retaining weak ones.
Insufficient Review Processes
Problematic items often remain in use simply because no one has evaluated them recently. Regular, systematic item review procedures prevent this type of oversight from damaging your exam.
Making the Retirement Decision
When you’ve identified a questionable item, take decisive action:
Promptly Remove the Item from Active Use
When you’ve detected a serious validity concern, flag the item as inactive immediately. Don’t wait for the next development cycle. If candidates have already been scored on the flawed item, consult with your psychometrician to determine whether rescoring is warranted and how to handle any affected pass/fail decisions.
Thoroughly Document the Decision
Create detailed records explaining why the item was retired, including statistical evidence, expert feedback, content concerns, and/or legal issues. This documentation is essential for demonstrating quality assurance and defending your program if decisions are questioned.
Conduct Root Cause Analysis
Examine what caused the item to fail. Common causes include flaws in writing, difficult-to-assess content areas, insufficient expert review, and inadequate pilot testing. Use failed items as learning opportunities to strengthen your development process.
Review Related Items
If one item has problems, check whether others testing similar content or using similar formats might have comparable issues. Consider reviewing questions developed by the same writer to see if patterns emerge.
Accountability and Transparency
Documentation demonstrates due diligence, supports continuous improvement, and provides evidence of quality assurance if your program faces scrutiny.
Document your item retirement processes and decisions carefully:
- Maintain records showing which items were retired, when, and why
- Track trends in retirement reasons to identify systemic issues
- Report retirement statistics to governance bodies as evidence of quality assurance
- Include item lifecycle management in your program’s standard operating procedures
- Train all team members on retirement criteria and processes
The Bottom Line: When to Retire Exam Items
Item retirement decisions should be driven by evidence, not convenience. When the data shows an item should be retired, act decisively. Your program’s integrity depends on it.
Don’t let problematic items undermine years of careful development. Quadterion provides the statistical analysis and expert guidance you need to make confident, defensible decisions on when to retire exam items. Contact us to see how we can support your quality assurance process.