Interobserver Agreement
IOA assesses whether two independent observers produce similar data when measuring the same behavior at the same time. High IOA (80%+) indicates that the measurement system is reliable and that the data can be trusted. Low IOA suggests problems with the operational definition, the measurement procedure, or observer training. IOA should be collected for at least 20-33% of sessions.
Procedural Integrity
Procedural integrity (also called treatment fidelity or procedural fidelity) measures the extent to which the intervention is implemented as designed. If the procedure is not implemented correctly, you cannot draw valid conclusions about its effectiveness. Measure procedural integrity by having an observer score whether each step of the procedure was implemented correctly, then calculate the percentage of steps implemented correctly.
Ensuring Accurate Measurement
Operationally define all target behaviors with observable, measurable criteria. Train observers to criterion (typically 80%+ IOA) before data collection begins. Conduct periodic reliability checks throughout the study or intervention. Provide booster training when IOA drops below acceptable levels. Use the simplest measurement system that captures the relevant dimension of the behavior.
Threats to Measurement Accuracy
Observer drift: observers gradually change how they apply the operational definition over time. Reactivity: the behavior of the individual changes because they know they are being observed. Complexity: trying to measure too many behaviors simultaneously reduces accuracy. Fatigue: long observation periods reduce observer attention and accuracy. Address these threats through regular training, unobtrusive observation methods, and manageable data collection schedules.