FREE NEWSLETTER: Econintersect sends a nightly newsletter highlighting news events of the day, and providing a summary of new articles posted on the website. Econintersect will not sell or pass your email address to others per our privacy policy. You can cancel this subscription at any time by selecting the unsubscribing link in the footer of each email.

posted on 15 February 2016

Are Citation Rates The Best Way To Assess The Impact Of Research?

from The Conversation

-- this post authored by Simon Chapman, University of Sydney

The United Kingdom's Medical Research Council (MRC) - the equivalent of Australia's National Health and Medical Research Council - recently released its 2014-15 economic impact report, details of which make for interesting reading.

Since 2006, research funded by the MRC resulted in more than 94,000 publications, 63,000 of which (67%) were peer-reviewed.

The traditional starting point for considering the scientific impact of research are its citations. This is how many other research papers and editorials subsequently cite a given paper into the future or in a given number of years since publication.

Different numbers of researchers are involved in different fields of research. So when a relatively small number of scientists work in a study area, even if they write a spectacularly important paper, it can still receive a small fraction of the citations a comparably important paper receives in an area where more researchers are involved.

It would be unreasonable and misleading to conclude that a leading researcher in a small field had less scientific impact than one in a big field. Also, as can be seen from this table, a paper that has been published for a short time has naturally had less time to be cited than one that has been out for longer.

Normalised citation impact

For these reasons, citation analysts use the normalised citation impact (NCI) to adjust citation volumes in different fields. This allows them to be compared in analyses of entire research funding schemes, national research and international activity.

The MRC report provides a graph showing the NCIs for all MRC-funded research publications for the sample period 2006-13. The average NCI citation for all papers in health and medical fields across these eight years was a desultory 2.08. And this was more than twice the world average.

The report notes that of more than six million papers, more than a fifth (21%) had not been cited, while only 3% of the MRC funded research had no citations to date.

What counts as a high and very high citation in the MRC data is also interesting. "Highly cited" means a paper with just four or more cites and "very highly cited" is anything with eight or more.

The situation in Australia is more opaque. A 2013 report provides lots of comparative data that show Australian health and medical researchers punch well above our weight compared to other nations and our small population.

But nowhere could I find comparable data to that provided by the MRC report. However, the NHMRC report notes:

The citation distribution among publications is very skewed. While very few publications achieve high citation counts, a vast majority receive very few or no citations at all.

Citation is of course only a measure of interest in your work by other researchers.

Journals that provide all or some of their papers as open access to anyone, often have daily updated counters showing the number of readers who have been to the publication. Not all of these will have read it and only land on it in a search for something else, so the data exaggerate actual readers.

My most read paper is one looking at the incidence of gun massacres and deaths ten years after the 1996 Australian gun law reforms.

It received some 136,946 views since 2006, 82,312 of which were in December 2012 in the aftermath of the Sandy Hook school shootings in the United States. I tweeted a link to the paper, that saw it go viral.

Ironically, the paper wasn't funded by any research grant.


A metric increasingly being used by researchers since 2013 to demonstrate wider interest in their work is Altmetric.

The Altmetric score provides an index of the extent to which a research paper is being circulated and discussed on social media and covered in the news media.

My 2006 firearms paper has a stratospheric Altmetric score of 2,118. The 100 highest Altmetric scoring papers across all research fields in 2015 are listed here. Had mine been published in 2015 with the same attention it has received it would have had the seventh highest Altmetric score that year.

The MRC report provides data on a range of social impacts that go well beyond metrics. The data assesses the direct impact of research on outcomes that may have social and economic impact.

These include:

  • The development of more than 4,400 instances of influence on policy and practice - 416 new in 2014, including 472 citations in clinical guidelines.

  • The development of more than 1,000 products and interventions - 126 new in 2014.

  • The creation or growth of 88 companies - seven new in 2014.

  • Approximately 1,081 patents - 37 filed or granted in 2014, with discoveries related to 246 (23%) of these patents already licensed worldwide.

In work we published in 2015 examining the "impact in society" of intervention research in health and medical research funded by the NHMRC, we found 38% of research projects could demonstrate some level of social or health service impact.

We investigated the characteristics of those projects that demonstrated impact and compared them to those that didn't.

Our study indicated that sophisticated approaches to intervention development, dissemination actions and translational efforts, are actually widespread among experienced researchers, and can achieve policy and practice impacts.

However, it was the links between the intervention results, further dissemination actions by researchers and a variety of contextual factors after the research that ultimately determined whether a study had policy and practice impacts.

Given the complicated interplay between various factors, there (alas) appears to be no simple formula for determining which intervention studies should be funded in order to achieve optimal policy and practice impacts.

The judgement of which research applications should be funded is, and is likely to remain, a very inexact science.

The ConversationSimon Chapman, Emeritus Professor in Public Health, University of Sydney

This article was originally published on The Conversation. Read the original article.

>>>>> Scroll down to view and make comments <<<<<<

Click here for Historical News Post Listing

Make a Comment

Econintersect wants your comments, data and opinion on the articles posted. You can also comment using Facebook directly using he comment block below.

Econintersect Contributors


Print this page or create a PDF file of this page
Print Friendly and PDF

The growing use of ad blocking software is creating a shortfall in covering our fixed expenses. Please consider a donation to Econintersect to allow continuing output of quality and balanced financial and economic news and analysis.

Take a look at what is going on inside of
Main Home
Analysis Blog
Was Marx Right?
Angst in America, Part 5: The Crisis We Can’t Muddle Through
News Blog
Guanxi: How Business Is Done In China
Measuring Interest Rate Risk In The Very Long Term
2016 Small Business Credit Survey: A Significant Majority Relied On Personal Finances
How Economic Changes Affect Congressional Budget Office's Budget Projections
Three Public Relations Blunders And How Your Company Can Learn From Them
Infographic Of The Day: Cheat Sheet: NAFTA's Mixed Track Record Since 1994
Early Headlines: Trump Blames Dems And Constitution For Chaos, US Child Poverty, Winter Leaves North New England, Labour Gains In Polls, And More
Grading President Trump's First 100 Days: B!
The Most Important Gaming Platforms 2017
Earnings And Economic Reports: Week Starting 01 May 2017
Ancestors Of Flores 'Hobbits' May Have Been Pioneers Of First 'Human' Migration Out Of Africa
The Global Top 10 Android Apps
What We Read Today 29 April 2017
Investing Blog
Technical Thoughts: Finding Contrarian Ideas
More People Have Access To Netflix Than A DVR
Opinion Blog
Investors: Super Size Me
How Our Addiction To Safety Could Lead To Another Financial Crisis
Precious Metals Blog
A New Age For Gold
Live Markets
28Apr2017 Market Close: Wall Street Closed Mostly Down On News The U.S. Economy Grew At Its Weakest Pace In Three Years, WTI Crude Settles In The Low 49 Handle
Amazon Books & More

.... and keep up with economic news using our dynamic economic newspapers with the largest international coverage on the internet
Asia / Pacific
Middle East / Africa
USA Government



Analysis Blog
News Blog
Investing Blog
Opinion Blog
Precious Metals Blog
Markets Blog
Video of the Day


Asia / Pacific
Middle East / Africa
USA Government

RSS Feeds / Social Media

Combined Econintersect Feed

Free Newsletter

Marketplace - Books & More

Economic Forecast

Content Contribution



  Top Economics Site Contributor TalkMarkets Contributor Finance Blogs Free PageRank Checker Active Search Results Google+

This Web Page by Steven Hansen ---- Copyright 2010 - 2017 Econintersect LLC - all rights reserved