Identifying and quantifying the unintended variability in common systematic observation instruments to measure youth physical activity

Background: Direct observation protocols may introduce variability in physical activity estimates. Methods: Thirty-five physical education lessons were video recorded and coded using the System for Observing Fitness Instruction Time (SOFIT). A multistep process examined variability in moderate to vigorous physical activity (MVPA%; walking + vigorous/total scans). Initially, per-SOFIT protocolMVPA% (MVPA%SOFIT) estimates were produced for each lesson. Second, true MVPA% (mean MVPA% of all students using all observations, MVPA%true) estimates were calculated. Third, MVPA% (MVPA%perm) was calculated based on all permutations of students and observation order. Fourth, physical education lessons were divided into 2 groups with 5 lessons from each group randomly selected 10,000 times. Group MVPA%perm differences between the 10 selected lessons were compared with the MVPA%true difference between group 1 and group 2. Results: Across all lessons, 10,212,600 permutations were possible (average 291,789 combinations per lesson; range = 73,440-570,024). Across lessons, the average absolute difference between MVPA%true and MVPA%SOFIT estimates was ±4.8% (range = 0.1%-17.5%). Permutations, based on students selected and observation order, indicated that the mean range of MVPA%perm estimates was 41.6% within a lesson (range = 29.8%-55.9%). Differences in MVPA% estimates between the randomly selected groups of lessons varied by 32.0%. Conclusion: MVPA% estimates from focal child observation should be interpreted with caution.