Jens Mittelbach

@jmiba.bsky.social

Head of Library Services at European University #Viadrina Frankfurt (Oder) @viadrina.bsky.social; Editor-in-Chief of #OA journal Bibliothek – Forschung und Praxis; #openness advocate; #3D aficionado; bicycle owner.

Could a trusted ‘novelty score’ change how we judge research? OpenAlex is considering adding an LLM-based novelty indicator alongside citations—transparent, reproducible, and field-wide—but only if the community wants it. Questionnaire here: #researchassessment blog.openalex.org/should-opena...

Should OpenAlex publish a novelty score? A vibe check

Kyle Demes: When people want to know whether a piece of research is any good, they usually have two options: ask an expert, or count citations. Both have well-known problems. Expert review is slow, expensive, subjective, and does not scale to portfolios of thousands of papers. Citations take years to accumulate, reward the already well-cited, and say more about whether a paper was used than whether it said anything new.That second gap is the interesting one. Creative work, including scientific discovery, is generally understood to be both new and useful. Citations are a reasonable proxy for usefulness. We have had no widely trusted, scalable measure for new. Funders say they want novel research, and then evaluate it with tools that cannot see novelty.Earlier this year an international challenge set out to change that, and the winning approach now exists as a working prototype. We are considering adding a novelty score to OpenAlex. Before we write a spec or scope resources to implement it, we want to know whether you would use it, what for, and what would make you trust it.The challengeThe Metascience Novelty Indicators Challenge was launched in September 2025 by the UK Metascience Unit at UK Research and Innovation; designed by the Science Policy Research Unit (SPRU) at the University of Sussex and RAND Europe; and operated by Challenge Works. Teams were asked to build a tool that could assess the novelty of a research paper at the time of publication.To evaluate the entries, SPRU built a ground-truth dataset the hard way: a random sample of recent publications across all disciplines (drawn from OpenAlex), each rated for novelty by human experts in the relevant field who were provided with the full text. Challenge entrants never saw those ratings. Their indicators were scored on how closely they matched the experts’, and the evaluators were blinded to who had submitted what.The winner, announced just a few months ago in June 2026, was a team from Forschungszentrum Jülich in Germany: Jan Göpfert, Samuel Kieling, Jann Weinand, Titan Hartono, and Patrick Kuckertz. Their approach came top on most evaluation criteria (in a “winner of winners approach”), and they received a £300,000 prize, provided by Coefficient Giving, to develop it further.Coverage: https://www.nature.com/articles/d41586-025-01882-7 and https://www.science.org/content/article/how-novel-research-paper-competition-quantify-concept-crowns-winner.How the Jülich indicator worksMost existing novelty indicators work from metadata: unusual combinations of references, keywords, or journals. The Jülich indicator reads the paper instead.It uses a large language model to analyse the study and a selection of the works it cites, and from those, it reconstructs the state of knowledge at the time of publication, including the open questions in the field. It then asks what the focal study contributes anew: an original method, a surprising result, or a solution to a problem that was previously unsolved. It deliberately collects arguments for and against the paper’s novelty and weighs them. The output is a score from 0 to 100, an interval showing how confident the model is, and a written justification that a reader can check.Jülich press release: https://www.fz-juelich.de/en/news/archive/press-release/2026/ai-identifies-scientific-novelty-julich-team-wins-international-challengeThe team is clear that this is a support for human judgement rather than a replacement for it, and that a score which could be gamed, or which quietly disadvantaged some fields or languages, would be worse than no score at all. Those are our concerns too.Why OpenAlex is interested, and what it would takeOpenAlex already carries citation counts, topics, and a growing set of work-level signals (e.g., field-weighted citation impact). A novelty score that is transparent, reproducible, and available for every work, free, in the same place, would be a genuinely new thing for funders, institutions, journals, and researchers to work with. The UK Metascience Unit has told us that it would be of interest to a large proportion of the roughly 100 research teams it funds, as well as to UKRI’s own analysts. And we would rather that such a novelty indicator be built in the open than watch it appear behind a paywall.It is not a small undertaking, and the open questions are the reason for this post:How far back? Scoring every work in OpenAlex is very different from scoring everything published since, say, 2019.Which works? Journal articles only, or preprints, conference papers, book chapters, and dissertations as well? The indicator was validated so far only on journal articles. Maybe that’s enough (after all, FWCI only applies to a few work types), but maybe it’s worth the effort to evaluate the indicator in a broader set of work types first. One-off or continuous? A static file scored once is much cheaper than a live score that updates as new works arrive and as records change. A live production database like OpenAlex really wants the latter, but maybe the community only needs static scores, updated yearly.Open-weight or commercial models? Open-weight LLMs would make the scores reproducible over the long term and could run on public research computing. Jülich is testing whether they perform as well.Cost and compute. Running a language model over hundreds of millions of works is a significant compute bill, however it is done. That’s why answers to the questions above are especially important before we fully spec this out.Transparency versus gaming. Publishing the full method makes the score checkable and makes it easier to write to. Where should the line sit and is that an argument for static historic application vs. on-going scoring of the indicator against new works?Use in evaluation. A score built to describe a literature can be misused to judge individuals. What safeguards would you want before it existed at all?What we are askingThis is a vibe check with a light-touch questionnaire, not a full consultation. We want enough signal on demand and scope to write a spec and a resourcing case, and we will go deeper later if the response warrants it.Fill in the form. About six questions, five minutes, any language. It stays open until Friday 13 November 2026.https://docs.google.com/forms/d/e/1FAIpQLScOlmbDN7AdJic8jHO1dnQWMhppFIrMK96J0s6TmcU1oME3ZQ/viewform?usp=publish-editorJoin the webinar. Wednesday 7 October 2026, 8:00 to 9:00am Pacific (11am Eastern, 4pm UK, 5pm Central Europe). Ben Steyn (UK Metascience Unit) on why the challenge was run, Jan Göpfert (Jülich) on how the indicator works, and Kyle Demes (OpenAlex) on what integrating it into OpenAlex would look like, followed by half an hour of questions. Sarah Otner (SPRU) will be on hand for questions about the evaluation. Recording and slides posted afterwards.https://zoom.us/webinar/register/WN_EetlA-l3QgK2FvQVxE0GLgPass it on. Funders, research offices, metascience groups, tool builders, and anyone who has ever wished that citations told them something they do not.What happens nextWe will share a synthesis of the responses with the UK Metascience Unit and the Jülich team in late November, and publish a summary here. If there is clear demand and a workable scope, the next step is a spec and a scoped first run. If there is not, we will say so and keep the door open.Thanks in advance for any input you share. If you have any other questions about this initiative, let me know at: kyle@openalex.org The post Should OpenAlex publish a novelty score? A vibe check appeared first on OpenAlex blog.

blog.openalex.org

Europe is drifting back towards the abyss. Rafael Behr warns how far-right parties, Maga ideology and Kremlin cash are aligning to smother liberal idealism—echoing the despair Stefan Zweig knew too well. #Europe #FarRight #Democracy www.theguardian.com/commentisfre...

European democracies are running out of time to turn away from the abyss | Rafael Behr

Rafael Behr: Far-right parties, Maga ideology and Kremlin money are conniving to suffocate liberal idealism for another generationNot all that long ago, Europe was a beacon of liberal idealism. One memoir recalls it as an era “comfortably saturated with confidence in the unfailing and binding power of tolerance and conciliation”. The economic system seemed obviously preferable to barbarous alternatives. “People honestly believed that the divergencies and the boundaries between nations and sects would gradually melt away into a common humanity,” the author added.Those words could describe the spirit of globalisation at the turn of the millennium. They were written in the 1930s by Stefan Zweig, reflecting on the optimism of late 19th-century Vienna from the perspective of a Jewish émigré whose world had been destroyed by murderous nationalism. Zweig took his own life in 1942. He never saw fascism defeated. He couldn’t have imagined a renaissance of liberalism in the second half of the 20th century so complete as to attain a pitch of complacency rivalling the one he remembered from his childhood. Continue reading...© Illustration: Anaïs Mims/Getty© Illustration: Anaïs Mims/Getty© Illustration: Anaïs Mims/Getty

theguardian.com

When “messy” affiliation data turns into a research security risk OpenAlex just cleaned 6,191 false links between authors and listed organisations—and they’re asking research security teams to help fix the rest. #ResearchSecurity #OpenAlex #Bibliometrics blog.openalex.org/when-affilia...

When affiliation errors become a research security problem

Kyle Demes: TL;DR. Research security screening checks author affiliations against lists of organizations of concern, using bibliometric data. Affiliation matching is imperfect, though, especially for companies and small institutes with short names or no ROR record, so a researcher can end up linked to an organization they have no connection to. As a first pass we cross-referenced the major watch lists to ROR and cleaned up 6,191 such over-merges across 192 listed organizations. We invite the research security community to collaborate with us to improve OpenAlex and use bibliometric data responsibly for this use case.Research security screening is now routine. Funders, universities, and commercial compliance tools check author affiliations against government lists of organizations of concern (e.g., entities flagged for military, defence, or sanctions reasons). The lists are public and the screening is increasingly required. One part of that screening process often relies on affiliation data from research publications. The general idea is to ask: has the research team been associated with organizations of concern directly (i.e., they worked for or were funded by them) or indirectly (through their networks of collaborators). A positive match in a bibliometric database then triggers follow-up scrutiny to determine if the connection is real and the risks associated with the connection. Anyone who has worked with affiliation data knows that it can get messy. To demonstrate the problem, the University of British Columbia has almost 200,000 distinct text strings that have been used in publications to refer to the University, covering departments, addresses, translations, typos, text encoding issues from publishers, and more. Here are a few examples:“The University of British Columbia”“UBC”“Univ of British Columbia, Vancouver, BC, Canada”“University of Brtish Columbia, Vancouver, Canada”“University of British Colombia#TAB#”And so algorithmic approaches are used to associate these text strings with institutions in the Research Organization Registry. Research information databases each have their own processes for doing this. We make our source code openly available, and you can find our methods detailed on github. But long story short, we have linked over 150M unique text strings to 130k unique institutions. These algorithms generally work well, with >98% precision and >90% recall for most academic institutions. But across the full corpus of all scholarly works in OpenAlex (500M ), that’s still millions of errors.These errors can be especially acute for companies or small research institutes because they may have very few publications to train the matching model and they often have short names that are common words. Consider the example of the company “Optical Sciences”. They have a ROR record because they publish research, but they get erroneously associated with the Optical Sciences Center of the University of Arizona which does not have its own ROR record.Most bibliometricians are accustomed to navigating these errors in their analyses, and some even know how to fix them upstream (e.g., curating ROR records and string matching in OpenAlex). On the other hand, research security professionals come from diverse backgrounds and only sometimes have a nuanced technical understanding of bibliometric databases. That means an affiliation error that used to be a minor data-quality annoyance can now hold up a grant, trigger a foreign-influence review, or put a researcher into a compliance inquiry over a relationship that never existed.A first pass at minimizing false positivesWe took the published watch lists that screening actually runs against (Canada’s Named Research Organizations, the US Department of Defense 1260H list, the US Consolidated Screening List including the BIS Entity List, US OFAC, the US UFLPA list, the UK FCDO sanctions list, and Australia’s DFAT list) and cross-referenced each entity to ROR with a strict country-consistency check. That produced a verified crosswalk of 192 organizations covering roughly 4.24 million works.Then, for each organization, we screened every affiliation string linked to it, about 2.5 million in all, looking for this specific failure: an unrelated company or institution absorbed into a listed entity because their names collide (often with an organization lacking a ROR record). We found 6,191 such over-merges, roughly a quarter of one percent of those strings, and we submitted manual curations to remove the linkages. The scale of the collision problem is easier to see in a few real examples than to describe.A small US biotechnology firm, identified on its papers by a short name only that also refers to a broad class of medical interventions, was linked to a Russian research institute whose registered acronym is nearly identical. The two have nothing to do with each other. The firm simply had no ROR record, so the closest-looking acronym won.System Equipments Div., Asahi Engineering Co., Ltd, a Japanese engineering company, was linked to a Chinese entity on the BIS Entity List whose registered name is essentially “System Equipment.”ENGINEERING-PHYSICS, INC. was linked to a Russian Institute of Engineering Physics that appears on these lists, on the strength of the shared words alone.None of these organizations has any connection to the listed entity it was linked to. In each case the two names are close enough that you can see why an algorithm reached for the wrong one because no record existed for the real organization. This work helps reduce these errors, but they still exist.Advice to analysts doing the screeningIf you’re an analyst in charge of screening research applications for potential connections with a list of organizations of concern, never assume that a bibliometric match is a definitive signal of a connection. A match can help you shorten your list of connections to confirm manually, but there are errors and you need to confirm any potential connections. You can do that easily by inspecting the raw_affiliation_string field in the OpenAlex API. For instance: https://api.openalex.org/works/w3134510327 -> raw_affiliation_string : “Optical Sciences, University of Arizona, Tucson, Arizona, USA”. You can also visit the publication’s landing page or PDF for more information directly from the source. All of that information is linked conveniently in the OpenAlex record.You should also take efforts to identify false negatives. For instance, if the organization of concern does not have a ROR record (or doesn’t have enough information on how the organization represents itself in publications), then we won’t be able to make any matches to that institution. More broadly, I highly recommend that you connect with experts in this field on how to successfully leverage bibliometric tools in your work while respecting their limitations. Advice to researchers submitting to screeningsTake a look in OpenAlex in advance to see if there might be issues in your author profile (my profile below). Sometimes another authors’ works might have snuck into your author profile which you can edit that there directly. And sometimes there are errors in affiliation attribution even when the work is yours (like in the examples above). When that happens, the quickest way to get it sorted out is to send details in an email to support@openalex.org. But keep in mind that funders and analysts use different databases and tools for research security compliance workflows and many of them use closed, proprietary databases you won’t be able to access. Let’s work on this togetherIf you work in research security, or at a funder or institution that screens affiliations, we want to hear from you.Screening decisions carry real weight, so the data behind them should be open to inspection. In OpenAlex every affiliation link traces back to the source publication, every correction is public and permanent, and the code that makes the matches is on GitHub (github.com/ourresearch). Nothing sits in a proprietary index that no one outside the vendor can check. When a match is wrong you can see why, contest it, and get it fixed, and that fix then helps everyone who screens after you. For a decision that can affect a researcher’s funding or reputation, that is not a small difference. Open access to the data also means scrutiny is not gated by budget: a well-resourced agency and an under-resourced one work from the same public record.We do not think we have this solved. We know some of the limits and we are certain there are more we have not found. What we want is to work with people who do this screening every day to map those limits honestly, so nobody treats a match as more certain than it is, and to build the documentation and tooling that would make an open, auditable corpus genuinely good infrastructure for this work. A few experts collaborating with us in the open could improve it for the whole community, globally.Get in touch at kyle@openalex.org.The post When affiliation errors become a research security problem appeared first on OpenAlex blog.

blog.openalex.org