Test Development Meeting Planning: A Comprehensive Contingency Guide for Facilitators

In an ideal world, all test development and delivery aspects would run smoothly for a credentialing organization. However, unforeseen circumstances always arise along the way. Some are minor inconveniences, like a subject matter expert (SME) being called into another meeting at the last second, or a laggy internet connection that prevents you from being on camera. Other situations, such as a burnt-out overhead projector bulb or a prolonged internet outage, require facilitators to adapt quickly. However, many of these issues can be mitigated through strategic test development meeting planning.

This article serves as a comprehensive guide to contingency planning for meeting facilitators using examples from psychometric test development sessions.

Why Test Development Meeting Planning Matters

Having a contingency plan is essential for high-stakes meetings, such as a job task analysis (JTA) or a standard setting study. Due to the critical nature of these sessions, which involve multiple SMEs, tight timelines and the expectation to meet all objectives, small disruptions can derail progress. The following is a step-by-step process that psychometricians, assessment developers and other facilitators can implement to ensure effective test development meeting planning that accounts for potential setbacks.

Step 1: Establish Your Meeting Environment

Before identifying the potential risks involved during your session, you must consider the environment where the meeting will occur. Test development meeting planning varies significantly based on format. Meetings can be held virtually through video conferencing platforms (e.g., Microsoft Teams, Zoom, Google Meet, Cisco Webex) or in person at client facilities or at a hotel or conference center. In some circumstances, a hybrid approach combining virtual and in-person modalities allows some participants to be remote and others to be in-person. The types of risks will vary depending on the setting.

Step 2: Identify Potential Risks in Test Development Meeting Planning

The categories identified below are a non-exhaustive list of potential risks that could postpone or derail the flow of a given meeting. These risks could be innocuous, or they could be severe. Effective test development meeting planning requires facilitators to prepare for any circumstance that might set back your meeting objectives and goals.

• Technology disruptions (e.g., choppy internet connections, laggy videoconferencing platforms, faulty electronics)
• SME availability (e.g., no-shows, technological interruptions)
• Test security (e.g., unsecured networks, inattention to printed or electronic materials, failure to set rules and procedures for accessing confidential material)

Managing Technology Disruptions

Whether virtually or in person, technology is embedded in our work. Video conferencing platforms can act up at inopportune times. Understanding the environment will alleviate any concerns about the meeting being derailed due to technology disruptions.

SME Technology Issues

Smart test development meeting planning includes proactive outreach to meeting participants about start times and the importance of logging onto meetings 5-10 minutes in advance so that SMEs can get comfortable with the platform or update any necessary software. SMEs with unreliable internet connections may need to turn off their cameras to participate in an uninterrupted meeting. Those with ongoing difficulty should also be given a backup, dial-in option to access if their issue is persistent. This should all be addressed before the meeting so that the scheduled activity can be used for its intended purpose.

For a systemic issue affecting all participants, being able to switch from one conferencing platform to another (e.g., from Teams to Zoom or Zoom to Google Meet) will prevent facilitators from having to cancel a meeting.

Facilitator Technology Backup Plans

If the facilitator has an internet outage during a virtual meeting, ensure that the activity can continue as intended by having a co-worker on the call concurrently or copying them on the meeting invitation so they can join after a brief pause to handle the meeting.

In-person meetings can also face technological problems that would force facilitators to resort to alternative options. Effective test development meeting planning includes discussing the layout of the meeting environment with the client ahead of time so that backup plans can be made. Will the meeting occur at the client’s facilities, or is it taking place at a hotel? If the plan is to use an overhead projector that casts the facilitator’s laptop screen for everyone to see, what happens if the projector malfunctions? A facilitator should identify other options in the room (e.g., whiteboard, easel or similar) or determine if other rooms are available. Accessing working files like content outlines or standard setting ratings from paper copies, a flash drive or your local desktop are good contingency plans for internet outages.

In a worst-case scenario, the facilitator should be proactive in adding additional time to meetings or scheduling an extra meeting if disruptions might cause delays in achieving objectives. Letting participants out early is much easier than asking them to stay beyond previously agreed-upon times!

Planning for Subject Matter Expert Availability

Due to unforeseen emergencies or last-minute circumstances, it is not uncommon to have an SME no-show. Comprehensive test development meeting planning should expect and plan that at least one SME will be unavailable. These situations can have damaging effects on the test development process that may impact the reliability or generalizability of findings and/or weaken the validity of evidence used to interpret test scores. For example:

Impact on Reliability: If fewer than six SMEs participate in the standard-setting group discussion, the recommended cut score will likely vary more and be less reliable than expected.

Impact on Generalizability: Many industries have a diverse panel of SMEs representing various demographics and professional experiences. Losing even one SME for any part of a JTA can call into question whether the test content outline adequately reflects the responsibilities, knowledge, and skills that a newly certified professional must demonstrate to perform safely on the job.

Impact on Validity: Losing SMEs during the JTA can result in a weakened validity argument because the test content outline may be more heavily focused on a specific area of the profession as opposed to a broad overview. There is less certainty that the cut score represents the minimum competency needed to perform a particular job, which may result in unqualified candidates passing or denying qualified candidates an opportunity to practice.

Facilitators should anticipate absences ahead of time. To alleviate any concerns about reaching an adequate number of SMEs, recruit one or two extra SMEs or have an extra on standby.

As mentioned in the previous section, facilitators should arrange an extra meeting or additional time to allow for scheduling flexibility in case too many absences potentially jeopardize the strength of our validity argument.

Test Security in Meeting Planning

The potential for an item or test to be compromised is persistent from the earliest stages of development, through delivery and ongoing psychometric analysis. Credentialing organizations should have a confidentiality/non-disclosure agreement (NDA) that all SMEs and other stakeholders interacting with test materials must sign to protect sensitive and proprietary information. Not only does an NDA serve as a legally binding agreement, but it also alerts SMEs to the high-stakes nature of the content. It sets ground rules for what is and is not acceptable when interacting with confidential information. Facilitators will reiterate the importance of the NDA at the inception of the meeting and remind SMEs throughout the process as needed. Test security requires ongoing monitoring that is beyond the scope of this article, but let’s discuss a few examples of potential threats facilitators should be aware of.

Job Task Analysis Security Considerations

Although specific test items are not usually discussed during JTAs, there is still the potential for sensitive information to be leaked. Facilitators must remind SMEs that the information being discussed is proprietary to the credentialing organization and that no content should be shared unless the sponsor consents. For example, some SMEs may expect specific content to become public and wish to discuss it through social media posts or word of mouth. As an outcome, an educational provider that views leaked content and accepts it at face value could begin to modify curricular activities in anticipation of these changes, only to realize later that the test sponsor released a final content outline without the leaked information included. This situation could lead to financial or reputational damages for the credentialing organization, SMEs and other stakeholders involved in the leak.

Standard Setting Security Protocols

Setting a cut score requires SMEs to evaluate items that have the most potential to be compromised, in-person or virtually. As meetings have shifted toward virtual formats, providing access to content has never been easier. Test links and passwords can be emailed to SMEs so they can access them in various settings and at any given time. However, even in-person meetings are still susceptible to item harvesting. For instance, meetings conducted at a hotel or other common space should always have a facilitator or client representative present in the activity room to prevent an item booklet from being stolen or to ensure that no one leaves their laptop open.

Regardless of format, a test form breach could have several negative consequences for stakeholders. Candidates may pass the exam because they memorized content instead of demonstrating competency, which allows unqualified candidates to pass. Test sponsors may incur additional costs and be forced to suspend testing until a new form is ready to launch. Additional reputational damage may erode the public’s faith in the credentialing organization and call into question the validity of the credential.

Facilitators should do the following to safeguard materials:

• Remind SMEs of their signed NDA
• Reiterate the confidential nature of the meeting and that all materials and discussions are considered confidential and proprietary
• Explain expectations and roles for SMEs, facilitators and other representatives
• Detail the negative consequences to all stakeholders in the event of a breach
• Keep tabs on all printed materials and set rules for virtual materials (e.g., passwords, expiration dates)
• Taking the above precautions is necessary to secure all materials and is an ongoing process that requires constant vigilance.

Conclusion: Making Test Development Meeting Planning a Priority

Potential risks could impede the progress of your meeting and undermine your assessment’s validity in many instances. Facilitators should always come prepared with contingency plans if something goes wrong.

The plans discussed above are some potential scenarios that could happen along the test development process. Facilitators should set expectations up front when working with clients and SMEs and be flexible when scheduling activities to anticipate potential setbacks. Effective test development meeting planning is an investment that pays dividends in successful outcomes and stakeholder satisfaction.

Quadterion leads credentialing organizations through the test development process by listening to your organization’s unique needs and being proactive in your program’s customized design and development. Reach out today to learn more about how we can supercharge your next JTA or standard setting meeting!