What Is Interobserver Agreement?
Interobserver agreement (IOA) is a measure of the degree to which two or more independent observers agree on the occurrence or measurement of a behavior. It is a critical component of behavioral research and practice because it provides evidence that the data are reliable. If two observers watching the same behavior at the same time record similar data, we can be more confident that the measurement system is working correctly.
IOA does not prove that the data are accurate (both observers could be making the same error), but high agreement suggests that the measurement procedures are clear and consistently applied. The BCBA exam frequently tests knowledge of IOA calculation methods, acceptable levels, and when to use each approach.
When to Collect IOA Data
IOA should be collected during a portion of sessions across all phases of a study or intervention. The generally accepted standard is that IOA should be assessed during at least 25% to 33% of sessions, distributed across all conditions and phases. Collecting IOA during only one phase (such as baseline only) does not adequately demonstrate the reliability of the measurement system.
Both observers must independently and simultaneously record the behavior without being able to see each other's data. If one observer can see the other's recordings, the data are not truly independent, and the IOA assessment is compromised.
Total Count IOA
Total count IOA is the simplest method. It compares the total counts recorded by each observer and calculates agreement as a percentage.
Formula: (Smaller count / Larger count) x 100
For example, if Observer A records 45 instances and Observer B records 50 instances, total count IOA = (45/50) x 100 = 90%.
Limitation: Total count IOA can be misleading because it does not account for whether observers agreed on the same specific instances. Two observers could record the same total number of responses while disagreeing on when those responses occurred. This is why more precise methods are often preferred.
Exact Agreement (Point-by-Point) IOA
Exact agreement IOA compares observer recordings on a point-by-point or moment-by-moment basis. The observation period is divided into smaller units (intervals, trials, or measurement occasions), and agreement is assessed for each unit.
Formula: (Number of agreements / (Number of agreements + Number of disagreements)) x 100
An agreement occurs when both observers record the same value for a given unit. A disagreement occurs when they record different values. This method provides a more accurate picture of agreement than total count IOA.
Trial-by-Trial IOA
Trial-by-trial IOA is used with discrete trial data. Each trial is compared between observers, and agreement is calculated across all trials.
For example, during a 20-trial session, Observer A and Observer B compare their recordings for each trial. If they agree on 18 of 20 trials, IOA = (18/20) x 100 = 90%.
This method is appropriate when behavior is measured as correct or incorrect responses across distinct learning trials.
Interval-by-Interval IOA
When interval recording is used (partial interval, whole interval, or momentary time sampling), IOA can be calculated on an interval-by-interval basis. Each interval is compared to determine whether both observers recorded the same result (occurrence or nonoccurrence).
Formula: (Number of intervals with agreement / Total number of intervals) x 100
This is a straightforward application of the exact agreement method to interval data.
Scored-Interval IOA
Scored-interval IOA (also called occurrence agreement) calculates agreement only for intervals in which at least one observer scored the behavior as occurring. This method is preferred when the behavior occurs infrequently (low-rate behavior) because standard interval-by-interval IOA can be artificially inflated by the many intervals in which both observers agree on nonoccurrence.
Formula: (Number of intervals with both observers scoring occurrence / Number of intervals with at least one observer scoring occurrence) x 100
Unscored-Interval IOA
Unscored-interval IOA (also called nonoccurrence agreement) calculates agreement only for intervals in which at least one observer scored the behavior as not occurring. This is preferred when the behavior occurs at high rates because standard interval-by-interval IOA could be inflated by many intervals of agreement on occurrence.
Formula: (Number of intervals with both observers scoring nonoccurrence / Number of intervals with at least one observer scoring nonoccurrence) x 100
Choosing the Right IOA Method
The choice of IOA method depends on the measurement system being used and the rate of the behavior:
- Frequency/event recording: Use total count IOA for a quick estimate, but exact agreement is preferred when possible.
- Discrete trial data: Use trial-by-trial IOA.
- Interval recording with low-rate behavior: Use scored-interval IOA to avoid inflated agreement from nonoccurrence intervals.
- Interval recording with high-rate behavior: Use unscored-interval IOA to avoid inflated agreement from occurrence intervals.
- General interval recording: Interval-by-interval IOA is acceptable when the behavior occurs at moderate rates.
Acceptable Levels of IOA
The generally accepted minimum standard for IOA in behavioral research is 80%, with 90% or higher considered good. However, the acceptable level may vary based on the complexity of the behavior being measured, the measurement system used, and the stakes involved. For high-stakes clinical decisions, higher IOA is especially important.
When IOA falls below acceptable levels, the behavior analyst should examine the operational definition of the target behavior, retrain observers, simplify the measurement system, or consider whether the behavior is too difficult to measure reliably with the current procedures.
Exam Tips
On the BCBA exam, be prepared to calculate IOA using given data, select the appropriate IOA method for a described scenario, and explain why IOA is important. Remember that IOA measures reliability (consistency), not validity (accuracy). Two observers can consistently agree on an incorrect measurement.