← Workflow data collection plan
Data collector calibration: a group practice guide
When more than one person records a workflow measure, a shared definition helps them apply the same rules. Use this exercise to uncover ambiguity and improve your collection instructions before a test.
Prepare the group session
Schedule a live, screen-shared and recorded group session. Confirm recording arrangements and accessible materials, explain recording access and retention to participants, follow your organization’s recording and retention procedures, and use fictional examples without identifying details. Bring the draft definition, eligibility rules, recording form and a few examples that include clear and ambiguous cases.
Run the exercise
- Read: Share the purpose, the definition, what counts as eligible, and how to mark an unknown answer.
- Classify: Have each collector apply the same rules to each fictional example independently before discussion.
- Compare: Discuss differences and what information each person used. Ask whether the definition or form needs clarification.
- Revise and repeat: Update the written instructions, record the version and date, and practice with new examples. As collection proceeds, compare a sample of entries with the source, review missing or unknown entries, and resolve discrepancies under the agreed process.
Use the discussion to improve the process, not to judge individual staff. Agreement on a few examples cannot establish overall reliability or accuracy.
Try three fictional cases
For this exercise only, an eligible handoff is one where the form is required. A delay means the form link was still not found two minutes after a search began. If the case does not say whether the link was found within two minutes, mark the event unknown; do not count unknown as no delay.
- The form is required. Search begins at 9:00 and the link is found at 9:01. What do you record?
- The form is required. Search begins at 9:00 and the link is still not found at 9:02. What do you record?
- The form is required, but the time the search began is not known. What do you record?
Suggested classifications
Case 1: eligible, no delay. Case 2: eligible, delay. Case 3: eligible, event unknown because the definition cannot be applied from the information given. In all three cases, the form is required; a case where it is not required would be ineligible under this exercise’s rule.
Record what your team learned
This worksheet has no save button. Print or copy your work to an approved document before leaving; your browser or printed files may retain copies. Do not enter names, client details or other identifying information.
Include eligibility, event definition, unknown handling and version.
Note where the group interpreted the same example differently.
Record changes to definitions, examples or the collection form.
Record the responsible role, date, new examples and follow-up check.
Next step
Use the workflow data collection plan to document the agreed method. Contact Special Learning to discuss group training for your organization.
Review your test together
Turn the evidence into a next step with the workflow test review agenda.
Sources
Informed by the Institute for Healthcare Improvement’s data collection planning guidance and the Agency for Healthcare Research and Quality’s guidance on quality measurement.
