Bibliometrics: Are we doing everything wrong?

A methodological dispute in the world of bibliometrics has thrown into question a high-profile, three-year-old study showing that science is getting less 'disruptive.'

One group says that no matter how you slice it, there is a smaller proportion of papers these days that are true breakaways from previous research. Most work, they say, is simply consolidating old ideas. But others counter that this apparent trend is largely an artefact of bad data and cultural changes in scientific publishing. By tweaking how you look at the question, some say, you can show that science is actually getting more disruptive.

The debate gets into the weeds of bibliometrics—the quantitative analysis of published literature—but it also touches on bigger questions: how open are important datasets, what's the best mechanism for moderating scientific differences, what is the point of science and how should we target scientific funding to optimize that? In short: are we doing everything wrong?

At the beginning of this saga, back in 2016, economic sociologist Russell Funk of the University of Minnesota and colleagues developed a new metric: the CD index (Funk, R.J. & Owen-Smith, J. Management Science 63(3):791-817 (2016)). The index is generated by looking at a given paper and asking: do future works cite this paper, but not the work that it built upon? That makes it 'destabilizing' or 'disruptive.' Or do future works cite both this paper and the works it references, making it more 'consolidating' of the status quo? Right from the start, Funk wondered if changes in the CD index over time might show that science is becoming less innovative, as some other papers had suggested. He and his colleagues crunched through 45 million papers to check.

The result, published in Nature in 2023, showed a precipitous 90-100% drop in the index from 1945 to 2010 across all fields, indicating a severe drop in disruptiveness (Park, M., Leahey, E. & Funk, R.J. Nature 613, 138-144 (2023)). "The finding was new but not necessarily that shocking to people that work in this space," says Funk. But it "just had this tremendously large reception, and that was something that definitely surprised me."

Stuck in a Rut?

The paper, which was chosen for the cover of Nature, made waves. It was read widely, referenced in policy reports, news stories and editorials, with plenty of hand-wringing about a crisis in declining innovation (see, for instance, "Is science getting less disruptive?" on the FQxI podcast). People speculated that researchers were getting stuck in a rut. Maybe the huge volume of published science makes it impossible for researchers to read widely, pushing them into narrower corridors of expertise and thought. Or maybe the system encourages them to publish many small papers, discouraging bigger, risky projects that might fail. Whatever the case, it didn't sound good.

This had some unexpected—and unwanted—repercussions. "Something I did not expect and has been unfortunate is the paper has been picked up by different politicians and policy groups to say: oh look, science isn't performing well, let's cut science budgets," he says. In hindsight, Funk says, they might have framed their results better. "If we knew so many people were going to read it, we would have said some things differently." Both consolidation and disruption are vital to science, he says; it's just that if the ratio between them is changing, that probably deserves some attention. "A healthy scientific ecosystem needs a balance."

The paper's reception in the world of bibliometrics was mixed. Some were impressed by the CD index and its clever way of using the citation network to get at something interesting. For instance, when speaking to the New York Times about the metric, Pierre Azoulay, an expert on technological innovation, entrepreneurship and strategic management at the Massachusetts Institute of Technology, in Cambridge, said, "I was giddy when I saw it. It's like a new toy."

But others questioned whether the analysis meant science was taking a nosedive in any real way. "There was quite a bit of initial, very quick pushback against this paper," says system scientist Alex Peterson at the University of California Merced. "My first reaction was mostly doubt about the construction of the indicator," says Vincent Larivière, an information scientist at the University of Montreal, in Quebec.

Various researchers have pointed out that is the disruption index is vulnerable to varying citation cultures across time and space, such as the prevalence of self-citation rings, or a trend towards publishing longer papers with more references. "Different social groups cite differently," says Larivière.

One study published in May (Cui, H. et al. Science 392, 588-591 (2026)) used the disruption index to show that researchers with a higher average academic age produce less disruptive output—a work that has been used to argue in favor of a mandatory retirement age. But another paper, posted in June, responded by showing that the disruption index is unusually high for researchers in Iraq and Nigeria, while India and China rank low, despite having similar academic ages of their research workforces (Larregue, J. & Gingras, Y. arXiv:2606.07332 (2026)). This suggests that the index is showing something else entirely—like particular characteristics of small, underfunded national research systems.

Dropped Data?

When Funk's paper came out, Vincent Holst was at the start of his PhD in mathematics at the Vrije Universiteit Brussel in Belgium and needed a project, his supervisor Vincent Ginnis suggested he dig into the CD index. What Holst found was surprising. First, there was a bug in the graphing software that Funk had used, which made it drop a bar from a chart. The bar it dropped was a weirdly anomalous spike of about a million papers at CD=1, the maximum and most disruptive category.

Holst wondered if this spike might be an accidental artefact caused by papers simply missing their references in the metadata. If a paper shows as having zero references, then of course future works cannot also cite those references, boosting its apparent disruptiveness to the max. Holst didn't have access to the same Web of Science dataset that Funk used, so he used a different one, SciSciNet: in this database, a suspiciously high 97% of papers with CD=1 showed zero references. In a spot check of 100 of these papers, 93% really did have references, they were just missing from the metadata.

When Holst dropped the odd CD=1 spike from the analysis, the trend all but disappeared. The apparent drop in disruption that Funk reported, Holst concluded, was driven in large part by a metadata mistake that was worse in older papers.

Holst posted his work on arXiv in February 2024; it then took years for the analysis to wind its way through the process of getting published in Nature as a 'matters arising' response to the original paper, alongside a reply from Funk's group (Holst, V., Algaba, A., Tori, F. et al. Nature 656, E7-E13 (2026) and Park, M., Leahey, E. & Funk, R.J. Nature 656, E14-E21 (2026)). The correspondence between the teams, both say, was frustrating, with unsatisfying email exchanges that never really resolved things. It's typical to take longer for a 'matter arising' to be published than the original paper, says Holst's co-author Sylvia Wenmackers, an FQxI member and philosopher of science at KU Leuven, Belgium, but she adds that such delays in handling controversies are "not a good thing for science in general."

Both of the Nature publications each pointedly highlight things that seem like obvious mistakes, like Funk dropping a million papers from a graph, and Holst using a dataset that includes a large number of non-academic works that of course have no references—notably including the Captain Underpants series of children's books. Both points are true, but arguably inflammatory. "There's a fair amount of argument by embarrassment here, which I don't think is professional," says John Drake, an ecologist with an abiding interest in science culture, who wrote a column about the issue for Forbes, in 2025.

In Funk's response, he argues that the problem of missing references in metadata is far less prevalent in Web of Science than in SciSciNet. Not all papers with CD=1 have zero references anyway, Funk adds, and some genuine breakthroughs really are papers with zero references. While Funk has no intuitive explanation for why there are so many papers with CD=1, he says "We knew that those papers were in there and we included them intentionally." But some observers, including Stephen Curry, a former molecular biologist and now advisor for the Research on Research Institute in London, UK, think the situation is odd: "When's the last time you read a paper with zero references?" says Curry.

Funk adds that even when using Holst's methodology, there is still a downfall in disruption: the size of the effect is much smaller than in the original paper, but still significant: comparable to, for example, the notable upward trend in team size over the same time period.

"It's kind of striking that they've been arguing this for three years, through an extensive process mediated by Nature, and at the end of the day it's still not really clear who's right," says Curry.

Wrong Dialogue

The debate "turns a lot on technicalities," says Drake. "In some respects, both authors have it wrong—or the dialogue is wrong. The naive concept is that there's this thing called disruption, and there's a way to measure it, and some ways are good and some ways are bad."

"If the data were made available, then others could have really tested it," Curry notes. Curry advocates for the use of open datasets, such as one called OpenAlex, launched in 2022. But every data set has its own foibles.

Lutz Bornmann, a sociologist of science at the administrative headquarters of the Max Planck Society in Munich, Germany, says for him the concept of the CD index is compelling, but his own work has shown a disappointingly low correlation between expert assessments of disruptiveness and CD scores. "What we need now is a meta-analysis," he says, to sort through differing results. His own recommendation is that disruption studies should only be done on papers with at least 5 or 10 references and citations. "The more citations and cited references, the more reliable the results will be," he says. When Holst limits his own analysis to papers with more than 10 references, he says, the CD index goes up over time, not down.

So, it remains unclear if the CD index really is declining and, if so, what that means. It's plausible, for example, that all the 'low hanging fruit' of science has already been discovered, making breakthroughs harder to come by. Alternatively, one could argue that new tools are opening up entire new areas of research with their own sets of easy and hard questions. Studies of patented drugs have shown declines, suggesting that medical innovation is declining. Yet in the world of technology, progress seems to be moving at a breakneck pace. What's really going on?

Others have tried to get at this question in different ways, looking for example at atypical reference combinations as a measure of a paper's novelty (Uzzi, B. et al. Science 342,468-472 (2013)). Some say that if you're going to stick with the CD index, it would be interesting to look at the absolute numbers of disruptive papers rather than the proportion. "The average paper is not disruptive. Disruptiveness is extremely rare. You want to focus on extreme values, not the average paper," says Peterson. Meanwhile, Peterson adds, other indicators show science getting 'better' in several regards: "If you look at productivity or other measures of progress, we're exponentially improving."

An AI-guided Exit from Citations

Some efforts try to break away from citations as a core measure altogether. Citations are handy because they are clear, countable numbers (how many papers cite your paper, for example, or how many citations a person racks up in their career) that are a valuable proxy for some aspects of quality. But they are also incomplete and fallible, prone to gaming, variable between disciplines and vulnerable to other quirks (older scientists, for example says Bornmann, tend to cite more and older references than early-career researchers).

Artificial intelligence might help to provide an exit from citation dependence in bibliometrics: large language models (LLMs), after all, can read a paper and assign it a score for novelty or disruptiveness or whatever one likes in a qualitative way, in the same way that expert peers might do, but for much larger samples. In 2025, the UK Metascience Unit and collaborators launched a novelty indicator challenge: quite a lot of the entrants, including the winner, used an LLM.

AI may be set to change the fundamentals we are seeking to measure, too. There's plenty of evidence that LLMs can help researchers to boost their efficiency on grunt work, for example, presumably giving them more time for thinking. And AI tools let researchers more efficiently find and digest a broader swath of research, which could boost interdisciplinary ideas. "Clearly there's tremendous opportunities for AI to stimulate disruptive work," says Funk. On the other hand, AI also has been shown to make the writing of papers and grant applications more similar to each other, as the bots mimic previous successful efforts.

Will AI help to broaden our horizons? Or will it narrow them? "There are arguments for and against," says Larivière. "The jury's still out."

All these terms used to describe the quality of research—innovation, disruption and novelty—are arguably fuzzy and hard to define, especially with a number. The rule of thumb known as Goodhart's Law always looms: when a measure (AI-generated or otherwise) becomes a target, it ceases to be a good measure. This makes plenty of experts wary of new metrics. "You should run away from single numbers all the time," says Larivière. "Especially bold claims based on a number that people don't understand.

Even Funk is ambivalent about single numbers as a stand-in for complex analyses of the quality of science. "I would welcome as much as anybody measures that didn't rely on citation data," he says. "They're a widely used proxy. They're useful for some things. But at the end of the day, they're kind of far from the thing that we really want to measure, which is ideas, knowledge themselves.

Lead image credit: gualtiero boffi.