Addressing unreliability propagation in scientific digital libraries

Todays scientists rely on scientific artifacts developed by others for their work. Individual scientists often have limited capacity to assess the validity of these resources. When errors are not caught, scientists produce second-generation errors. We say that a publication propagates unreliability when the main contribution of the publication becomes unreliable by using an unreliable source. An approach for checking whether publications propagate unreliability should satisfy three requirements, in priority order: (1) not miss any publications that propagate unreliability; (2) provide rationales; and (3) identify all publications that do not propagate unreliability. We consider three approaches: A base approach using metadata of the citing publications and the section headings of the citation contexts; and supplementing the base approach with either keyword-based or machine-learning-based modules. The base approach is the most generalizable. Approach-KW (base+keyword) provides concrete rationales, which could be important for convincing authors and editors to take action to update publications that propagate unreliability. Approach-ML (base+machine learning) has the best performance. Future work should develop a more general framework using multiple case studies. We will build a human-inthe-loop alerting system that digital library maintainers, editors, and authors could use to triage publications that may propagate unreliability, and maintain the quality of scientific digital libraries.