Why Interobserver Agreement Matters
Interobserver agreement (IOA) is a measure of the extent to which two or more independent observers record the same values when observing the same behavior at the same time. In applied behavior analysis, IOA data are essential for establishing the reliability and believability of behavioral measurement. Without IOA, there is no way to verify that the data being collected actually reflect the behavior occurring, rather than observer error, bias, or inconsistency.
The BACB Ethics Code and best practices in ABA research and clinical work require regular IOA assessments. Most standards recommend collecting IOA data for at least 30% of sessions, with agreement values of 80% or above considered acceptable for most applications, and 90% or above for research.
When to Collect IOA Data
IOA should be collected across all phases and conditions of a study or intervention program. Collecting IOA only during baseline or only during treatment can mask measurement drift or inconsistency that occurs in other phases. Best practice involves distributing IOA sessions across the entire duration of data collection.
IOA is especially important when training new observers, when operational definitions are complex, when subjective judgment is required in scoring, and when the data will be used for high-stakes clinical or research decisions.
Types of IOA
Total Count IOA
Total count IOA is the simplest form. It compares the total count recorded by each observer for the entire observation session. The formula is:
Total Count IOA = (Smaller Count / Larger Count) x 100
For example, if Observer A records 18 instances and Observer B records 20 instances, total count IOA = (18/20) x 100 = 90%.
The advantage of total count IOA is its simplicity. The disadvantage is that it can overestimate agreement because two observers might record the same total but at different times during the session. Observer A might record 10 instances in the first half and 8 in the second half, while Observer B records 5 in the first half and 15 in the second half. Both totals are close, but the observers were recording different events.
Mean Count per Interval IOA (Partitioned IOA)
To address the limitations of total count IOA, the observation session can be divided into smaller intervals, and IOA is calculated for each interval using the total count formula. The mean across all intervals provides a more accurate picture of agreement. This approach reveals disagreements that total count IOA would hide.
Exact Agreement IOA (Per-Interval)
Exact agreement IOA divides the session into intervals and scores each interval as either an agreement (both observers recorded the same value) or a disagreement (different values). The formula is:
Exact Agreement IOA = (Agreements / (Agreements + Disagreements)) x 100
This is the most stringent form of count-based IOA because it requires the observers to record identical values within each interval.
Trial-by-Trial IOA
Trial-by-trial IOA is used when data are collected on discrete trials, such as during teaching sessions. Each trial is scored as an agreement or disagreement, and the formula matches the exact agreement formula above. If there are 20 trials and observers agree on 18 of them, trial-by-trial IOA = (18/20) x 100 = 90%.
Scored-Interval IOA
When using interval recording or time sampling, scored-interval IOA focuses only on intervals in which at least one observer scored the behavior as occurring. This is particularly useful when the behavior occurs at low rates, because unscored intervals (where both observers agree the behavior did not occur) can inflate overall agreement.
Scored-Interval IOA = (Agreements on occurrence / (Agreements on occurrence + Disagreements)) x 100
Unscored-Interval IOA
Conversely, unscored-interval IOA focuses on intervals where at least one observer scored the behavior as not occurring. This is useful when the behavior occurs at high rates and you want to ensure observers agree on when the behavior is absent.
Duration IOA
When measuring duration, IOA compares the total duration recorded by each observer. The formula parallels total count IOA:
Duration IOA = (Shorter Duration / Longer Duration) x 100
Selecting the Right IOA Method
The choice of IOA method should match the measurement system being used.
- Frequency/event recording: Use mean count per interval IOA or exact agreement IOA for the most accurate results. Total count IOA is acceptable only when occurrence is relatively evenly distributed.
- Interval recording or time sampling: Use scored-interval IOA for low-rate behaviors and unscored-interval IOA for high-rate behaviors. Report both when possible.
- Discrete trial data: Use trial-by-trial IOA.
- Duration recording: Use duration IOA, preferably calculated per occurrence rather than as a session total.
Common Sources of Disagreement
Understanding why observers disagree helps clinicians improve measurement systems. Common sources include unclear operational definitions, insufficient observer training, observer drift over time, complexity of the behavior being measured, difficulty seeing or hearing the behavior, and recording system errors.
When IOA falls below acceptable levels, the first step should be reviewing and clarifying the operational definition. If the definition is clear but disagreements persist, additional observer training with practice and feedback is warranted.
BCBA Exam Application
The BCBA exam tests your ability to select the appropriate IOA method for a given scenario and to calculate IOA from provided data. Practice calculating each type of IOA until you can do so quickly and accurately. Be prepared for questions that present a measurement scenario and ask which IOA method would be most appropriate or which would provide the most conservative estimate of agreement.