FREE NEWSLETTER: Econintersect sends a nightly newsletter highlighting news events of the day, and providing a summary of new articles posted on the website. Econintersect will not sell or pass your email address to others per our privacy policy. You can cancel this subscription at any time by selecting the unsubscribing link in the footer of each email.

posted on 15 February 2016

Are Citation Rates The Best Way To Assess The Impact Of Research?

from The Conversation

-- this post authored by Simon Chapman, University of Sydney

The United Kingdom's Medical Research Council (MRC) - the equivalent of Australia's National Health and Medical Research Council - recently released its 2014-15 economic impact report, details of which make for interesting reading.

Since 2006, research funded by the MRC resulted in more than 94,000 publications, 63,000 of which (67%) were peer-reviewed.

The traditional starting point for considering the scientific impact of research are its citations. This is how many other research papers and editorials subsequently cite a given paper into the future or in a given number of years since publication.

Different numbers of researchers are involved in different fields of research. So when a relatively small number of scientists work in a study area, even if they write a spectacularly important paper, it can still receive a small fraction of the citations a comparably important paper receives in an area where more researchers are involved.

It would be unreasonable and misleading to conclude that a leading researcher in a small field had less scientific impact than one in a big field. Also, as can be seen from this table, a paper that has been published for a short time has naturally had less time to be cited than one that has been out for longer.

Normalised citation impact

For these reasons, citation analysts use the normalised citation impact (NCI) to adjust citation volumes in different fields. This allows them to be compared in analyses of entire research funding schemes, national research and international activity.

The MRC report provides a graph showing the NCIs for all MRC-funded research publications for the sample period 2006-13. The average NCI citation for all papers in health and medical fields across these eight years was a desultory 2.08. And this was more than twice the world average.

The report notes that of more than six million papers, more than a fifth (21%) had not been cited, while only 3% of the MRC funded research had no citations to date.

What counts as a high and very high citation in the MRC data is also interesting. "Highly cited" means a paper with just four or more cites and "very highly cited" is anything with eight or more.

The situation in Australia is more opaque. A 2013 report provides lots of comparative data that show Australian health and medical researchers punch well above our weight compared to other nations and our small population.

But nowhere could I find comparable data to that provided by the MRC report. However, the NHMRC report notes:

The citation distribution among publications is very skewed. While very few publications achieve high citation counts, a vast majority receive very few or no citations at all.

Citation is of course only a measure of interest in your work by other researchers.

Journals that provide all or some of their papers as open access to anyone, often have daily updated counters showing the number of readers who have been to the publication. Not all of these will have read it and only land on it in a search for something else, so the data exaggerate actual readers.

My most read paper is one looking at the incidence of gun massacres and deaths ten years after the 1996 Australian gun law reforms.

It received some 136,946 views since 2006, 82,312 of which were in December 2012 in the aftermath of the Sandy Hook school shootings in the United States. I tweeted a link to the paper, that saw it go viral.

Ironically, the paper wasn't funded by any research grant.


A metric increasingly being used by researchers since 2013 to demonstrate wider interest in their work is Altmetric.

The Altmetric score provides an index of the extent to which a research paper is being circulated and discussed on social media and covered in the news media.

My 2006 firearms paper has a stratospheric Altmetric score of 2,118. The 100 highest Altmetric scoring papers across all research fields in 2015 are listed here. Had mine been published in 2015 with the same attention it has received it would have had the seventh highest Altmetric score that year.

The MRC report provides data on a range of social impacts that go well beyond metrics. The data assesses the direct impact of research on outcomes that may have social and economic impact.

These include:

  • The development of more than 4,400 instances of influence on policy and practice - 416 new in 2014, including 472 citations in clinical guidelines.

  • The development of more than 1,000 products and interventions - 126 new in 2014.

  • The creation or growth of 88 companies - seven new in 2014.

  • Approximately 1,081 patents - 37 filed or granted in 2014, with discoveries related to 246 (23%) of these patents already licensed worldwide.

In work we published in 2015 examining the "impact in society" of intervention research in health and medical research funded by the NHMRC, we found 38% of research projects could demonstrate some level of social or health service impact.

We investigated the characteristics of those projects that demonstrated impact and compared them to those that didn't.

Our study indicated that sophisticated approaches to intervention development, dissemination actions and translational efforts, are actually widespread among experienced researchers, and can achieve policy and practice impacts.

However, it was the links between the intervention results, further dissemination actions by researchers and a variety of contextual factors after the research that ultimately determined whether a study had policy and practice impacts.

Given the complicated interplay between various factors, there (alas) appears to be no simple formula for determining which intervention studies should be funded in order to achieve optimal policy and practice impacts.

The judgement of which research applications should be funded is, and is likely to remain, a very inexact science.

The ConversationSimon Chapman, Emeritus Professor in Public Health, University of Sydney

This article was originally published on The Conversation. Read the original article.

>>>>> Scroll down to view and make comments <<<<<<

Click here for Historical News Post Listing

Make a Comment

Econintersect wants your comments, data and opinion on the articles posted.  As the internet is a "war zone" of trolls, hackers and spammers - Econintersect must balance its defences against ease of commenting.  We have joined with Livefyre to manage our comment streams.

To comment, using Livefyre just click the "Sign In" button at the top-left corner of the comment box below. You can create a commenting account using your favorite social network such as Twitter, Facebook, Google+, LinkedIn or Open ID - or open a Livefyre account using your email address.

You can also comment using Facebook directly using he comment block below.

Econintersect Contributors


Print this page or create a PDF file of this page
Print Friendly and PDF

The growing use of ad blocking software is creating a shortfall in covering our fixed expenses. Please consider a donation to Econintersect to allow continuing output of quality and balanced financial and economic news and analysis.

Take a look at what is going on inside of
Main Home
Analysis Blog
A Short Note on a Connection Between Marginalist Economics and Folk Medicine
Run A High Pressure Economy? Janet Yellen Does Not Understand the Problem
News Blog
How 'Neurosexism' Is Holding Back Gender Equality - And Science Itself
High Housing Costs Hinder Job Creation
What We Read Today 28 October 2016
Ten Ways To Live A Happier Life According To Animals
21 October 2016: ECRI's WLI Growth Index Again Declines
Advance Estimate 3Q2016 GDP Quarter-over-Quarter Growth at 2.9 Percent.
Rail Week Ending 22 October 2016 Better Than The Previous Week
What Happens After The Islamic State Loses Mosul
Infographic Of The Day: The History Of Women's Ice Hockey In Canada
Early Headlines: Asia Stocks Mixed, Huge Antarctic Marine Park, Can Trump Get To 270?, US Workers Gaining, UK Inflation, France GDP, India Savings Lag And More
Why Amazon Gives So Many Perks To Prime Members
Where Workplace Trust Is Strongest
How A Lack Of Sleep Affects Your Brain - And Personality
Investing Blog Weekly Summary 28 October 2016
Technical Thoughts: Looking For The Rebounds
Opinion Blog
Global Debt Investors: The Silence Of The Lambs
A Hard Brexit And Reduced Migration Won't Benefit UK Workers
Precious Metals Blog
Inflation Surging As Platinum Signals Stock Market Decline
Live Markets
28Oct2016 Market Close: US Stock Markets Close Fractionally Lower After FBI Opens Probe Into Hillary's Emails Over Shadows Positive GDP Report, Indicators Fractionally Bearish
Amazon Books & More

.... and keep up with economic news using our dynamic economic newspapers with the largest international coverage on the internet
Asia / Pacific
Middle East / Africa
USA Government

Crowdfunding ....



Analysis Blog
News Blog
Investing Blog
Opinion Blog
Precious Metals Blog
Markets Blog
Video of the Day


Asia / Pacific
Middle East / Africa
USA Government

RSS Feeds / Social Media

Combined Econintersect Feed

Free Newsletter

Marketplace - Books & More

Economic Forecast

Content Contribution



  Top Economics Site Contributor TalkMarkets Contributor Finance Blogs Free PageRank Checker Active Search Results Google+

This Web Page by Steven Hansen ---- Copyright 2010 - 2016 Econintersect LLC - all rights reserved