FCCC LOGO Faculty Publications
Handorf EA , Heckman CJ , Darlow S , Slifker M , Ritterband L
A hierarchical clustering approach to identify repeated enrollments in web survey data
PLoS One. 2018 Sep;13(9) :e0204394
PMID: 30252908    PMCID: PMC6155511     URL: https://www.ncbi.nlm.nih.gov/pubmed/30252908
Back to previous list
INTRODUCTION: Online surveys are a valuable tool for social science research, but the perceived anonymity provided by online administration may lead to problematic behaviors from study participants. Particularly, if a study offers incentives, some participants may attempt to enroll multiple times. We propose a method to identify clusters of non-independent enrollments in a web-based study, motivated by an analysis of survey data which tests the effectiveness of an online skin-cancer risk reduction program. METHODS: To identify groups of enrollments, we used a hierarchical clustering algorithm based on the Euclidean distance matrix formed by participant responses to a series of Likert-type eligibility questions. We then systematically identified clusters that are unusual in terms of both size and similarity, by repeatedly simulating datasets from the empirical distribution of responses under the assumption of independent enrollments. By performing the clustering algorithm on the simulated datasets, we determined the distribution of cluster size and similarity under independence, which is then used to identify groups of outliers in the observed data. Next, we assessed 12 other quality indicators, including previously proposed and study-specific measures. We summarized the quality measures by cluster membership, and compared the cluster groupings to those found when using the quality indicators with latent class modeling. RESULTS AND CONCLUSIONS: When we excluded the clustered enrollments and/or lower-quality latent classes from the analysis of study outcomes, the estimates of the intervention effect were larger. This demonstrates how including repeat or low quality participants can introduce bias into a web-based study. As much as is possible, web-based surveys should be designed to verify participant quality. Our method can be used to verify survey quality and identify problematic groups of enrollments when necessary.
1932-6203 Handorf, Elizabeth A ORCID: http://orcid.org/0000-0003-0445-8978 Heckman, Carolyn J Darlow, Susan Slifker, Michael Ritterband, Lee Journal Article United States PLoS One. 2018 Sep 25;13(9):e0204394. doi: 10.1371/journal.pone.0204394. eCollection 2018.