Image processing for public health surveillance of tobacco point-of-sale advertising: Machine learning-based methodology

Ned English, Andrew Anesetti-Rothermel, Chang Zhao, Andrew Latterner, Adam F. Benson, Peter Herman, Sherry Emery, Jordan Schneider, Shyanika W. Rose, Minal Patel, Barbara A. Schillo

Research output: Contribution to journalArticlepeer-review

Abstract

Background: With a rapidly evolving tobacco retail environment, it is increasingly necessary to understand the point-of-sale (POS) advertising environment as part of tobacco surveillance and control. Advances in machine learning and image processing suggest the ability for more efficient and nuanced data capture than previously available. Objective: The study aims to use machine learning algorithms to discover the presence of tobacco advertising in photographs of tobacco POS advertising and their location in the photograph. Methods: We first collected images of the interiors of tobacco retailers in West Virginia and the District of Columbia during 2016 and 2018. The clearest photographs were selected and used to create a training and test data set. We then used a pretrained image classification network model, Inception V3, to discover the presence of tobacco logos and a unified object detection system, You Only Look Once V3, to identify logo locations. Results: Our model was successful in identifying the presence of advertising within images, with a classification accuracy of over 75% for 8 of the 42 brands. Discovering the location of logos within a given photograph was more challenging because of the relatively small training data set, resulting in a mean average precision score of 0.72 and an intersection over union score of 0.62. Conclusions: Our research provides preliminary evidence for a novel methodological approach that tobacco researchers and other public health practitioners can apply in the collection and processing of data for tobacco or other POS surveillance efforts. The resulting surveillance information can inform policy adoption, implementation, and enforcement. Limitations notwithstanding, our analysis shows the promise of using machine learning as part of a suite of tools to understand the tobacco retail environment, make policy recommendations, and design public health interventions at the municipal or other jurisdictional scale.

Original languageEnglish
Article number24408
JournalJournal of Medical Internet Research
Volume23
Issue number8
DOIs
StatePublished - Aug 2021

Bibliographical note

Funding Information:
This work was supported in part by Truth Initiative and included activities such as data collection, analysis, interpretation of data, and writing the manuscript. In addition, any store photos located in Washington, DC, that were used in this research were collected from a study supported by the National Cancer Institute of the National Institutes of Health under award number R21CA208206. Although AAR is a United States Food and Drug Administration’s Center for Tobacco Products employee, this work was not done as part of his official duties. This publication reflects the views of the author and should not be construed as reflecting the United States Food and Drug Administration’s Center for Tobacco Products’ views or policies.

Publisher Copyright:
© 2021 Ned English, Andrew Anesetti-Rothermel, Chang Zhao, Andrew Latterner, Adam F Benson, Peter Herman, Sherry Emery, Jordan Schneider, Shyanika W Rose, Minal Patel, Barbara A Schillo.

Keywords

  • Convolutional neural network
  • Crowdsourcing
  • Image classification
  • Machine learning
  • Object detection
  • Public health surveillance
  • Tobacco point of sale

ASJC Scopus subject areas

  • Health Informatics

Fingerprint

Dive into the research topics of 'Image processing for public health surveillance of tobacco point-of-sale advertising: Machine learning-based methodology'. Together they form a unique fingerprint.

Cite this