You don’t know what you don’t know: how Signals helps publishers understand research integrity in their journals

One of the most interesting parts of building Signals is that we have a comprehensive view of research integrity across the publishing landscape, and can pinpoint specific issues impacting journals and publishers. The Signals Data Graph, which networks billions of nodes of publication data, enables us to see which journals have no or few research integrity cases, and spot journals that show signals of being the target of publication fraud. These research integrity issues are often undetected — it could even appear that a journal facing these issues is performing well, with increasing publications, citations, and a growing Journal Impact Factor.

We can see how Signals journal-level evaluation works through a tale of two (anonymised) journals. 

Journal A publishes hundreds of articles per year, it is indexed in Web of Science and Scopus and is an example of a journal with high standards of editorial rigor and research integrity. Signals evaluation of the journal looks like this:

Almost all of the articles in journal A ‘look good’ as we have not identified any significant issues. There is no sign of systematic issues impacting the journals. However, there are a small number of articles that are evaluated as ‘Caution’ and ‘Alert’. Looking closer at the ‘Alert’ articles, we can see that certain authors have a history of retractions due to image manipulation and other misconduct. Several articles also cite retracted articles, many of which are retracted for misconduct. As even small numbers of problematic articles can damage the reputation of otherwise high-quality journals, editors can use this information to address current risks and prevent similar cases in the future.

Journal B is an open access journal that publishes medical research. The journal is growing and similarly to journal A, it is indexed in Web of Science and Scopus. However, this journal has significant research integrity issues:

Journal B has an unusually high volume of articles with ‘Alert’ evaluations, and the types of issues and patterns that we see are indicative of papermill activity in the journal: 

• Many of these articles reference papermill articles — this is a good indicator of fraud, as papermills sell citations to their articles. 

• Almost half of the publications are by authors from institutions with high retraction rates.

• Over 70% of articles do not cite their authors — this is unusual, but it makes sense if we consider that many authors may not have been involved in the writing if they had paid for their authorship. 

The distribution of article evaluations that we see in journal B is similar to that of Hindawi journals between 2021-2023, and could be the result of papermills targeting gaps in editorial workflows or policies. 

We’ve all seen what can happen if a publisher is unable to identify and address these issues: delisting from journal indexes, a drop in high-quality submissions, a reduction in revenue, and reputational damage. There will also be a negative effect on researchers who have published legitimate articles in the journal. The impact could go beyond the publishers and authors; the problematic, and potentially fake, articles could waste the valuable time of researchers and clinicians, or worse, have real-world clinical consequences. 

To effectively protect their journals’ reputations and maintain trust in the scholarly record, publishers must first understand past and current integrity issues, as well as those on the horizon. Only then can publishers begin to identify the right approach for their research integrity operations and strategy. 

To meet this need, Signals provides custom reports that provide insight across a portfolio, from high-level summaries to detailed evidence at the journal and article level. We work with research integrity teams to incorporate their valuable expertise into our reports and help publishers integrate these insights into their workflows.