New in Signals: assess the relevance of references in manuscript submissions

Signals can now assess whether the references in a manuscript are actually relevant to the claims they’re attached to. It’s the next evolution of our reference checks, and a major step forward in helping editorial teams quickly evaluate manuscript quality.

Earlier this year, we launched the invalid references signal to catch hallucinated references, invalid DOIs, and metadata mismatches. But a valid reference can still have nothing to do with a claim — for example, a sentence about quantum computing backed by a citation to an agriculture paper. Irrelevant references happen through honest error, citation manipulation, or misuse of AI. Whichever way, it distorts citation metrics and is very difficult to catch when an editor is evaluating a manuscript with a long reference list under time pressure.

What editorial teams see

For each valid reference, Signals returns one of four relevance signals:

• Relevant — the cited paper is relevant to the claim it’s attached to

• Ambiguous — some overlap, but unclear whether the reference is relevant to the specific claim

• Irrelevant — no real connection, possible citation padding or coercion

• Not assessable — insufficient metadata available to assess the reference relevance 

This signal is accompanied by the relevance score, citation context, relevance details, and supporting scores, including citation network overlap. Behind the scenes, Signals combines citation context extraction, networked publication data, and LLM analysis to produce each signal.

Introducing Preview Signals

Reference Relevance is launching as our first Preview Signal — a way for us to roll out new signals. Users see the signal and its details, but it doesn’t yet have a score or affect the overall manuscript evaluation. Publisher workflows aren’t disrupted, and editorial teams can see the signal in action, share feedback, and help us refine it before it influences the manuscript evaluation.

Available now

The reference relevance signal is available now as a Preview Signal to all publishers using Signals Manuscript Checks via ScholarOne, Editorial Manager, eJP, bespoke integrations, and Direct Upload.

Book a demo to see how Signals helps your editorial teams catch the citation issues they’re missing today — and protect the integrity of your journals.