In an era of amazing advances in healthcare it’s easy to be dazzled by new scientific and technological developments, such as pharmacogenomics and the 3D printing of body parts.
But sometimes it’s the almost invisible innovations, for example, new approaches to evidence synthesis that can make the greatest contribution to health care.
In the following piece, Emily Steele, Anna Noel-Storr and Tari Turner, discuss two innovative new projects which use crowd-sourcing and open access platforms to support clinical decision making that is informed by the latest scientific evidence.
Emily Steele, Anna Noel-Storr and Tari Turner write:
The revolution began decades ago when Archie Cochrane called for ‘a critical summary, by specialty or subspecialty, updated periodically, of all relevant randomised controlled trials’, spearheading the systematic review (SR) movement and irrevocably changing the nature of healthcare.
As many readers know, SRs rigorously synthesise available evidence for a healthcare practice. Doctors, nurses, allied health practitioners, patients and policymakers can consult SRs for an unbiased summary of research evidence supporting a treatment.
Limitations to systematic reviews
However, there are weaknesses in the SR approach. For example, SR literature searches are often more than 12 months old at the time of SR publication, and trials published in that period won’t have been included. At least 75 health care trials are published daily, so that’s a lot of available evidence not incorporated in new reviews.
This culminates in clinicians and policymakers drawing on reviews that may be outdated and possibly misleading.
Project Transform, funded by NHMRC and Cochrane, is exploring solutions to this issue. The project aims to design, trial and scale-up more efficient SR production systems through collaborative networks and technology.
Two components of the project have been launched as open-access platforms and are well on the way to making a difference.
Cochrane Crowd – citizen scientists help curate and deliver health evidence
Cochrane Crowd is Cochrane’s new citizen science platform. Through Cochrane Crowd, a global community of volunteers help find the research needed to support informed decision-making.
Anyone can sign up to Crowd, undertake a short training program, and screen research studies to identify randomised controlled trials (RCTs), the gold-standard study design for testing health treatments.
Once identified by the Crowd, RCTs are deposited in a register of trials that can be accessed through the Cochrane Library by SR authors around the world, to find the evidence they need for their reviews. Through the efforts of the Crowd, RCTs are being identified quickly and accurately so that review authors can focus on other tasks critical to review production.
A promising start
Crowd was launched in March 2016 and has almost 4,600 contributors from 88 countries, with 111 Australians having joined so far. The Crowd has screened over 350,000 records and identified more than 32,000 RCTs – and the figures are ever-rising.
Cochrane Crowd is proving not only beneficial for review production, but a win for contributors too. Jannie Leung, Research Fellow and Epidemiologist at the University of Queensland, says
[quote]“I was initially interested in Cochrane Crowd because it offered training on screening abstracts for study design, and I thought it would be a good way to upskill. I found the training clear and accessible, and I learnt more about study design through the exercises. Also, there’s no need to commit big chunks of time – I go into Crowd for ten or fifteen minutes at a time, in between meetings or activities.”[/quote]
With the success of Cochrane Crowd to date, the platform is expanding. A new task for crowdsourcing the extraction of RCT details will launch soon, further streamlining SR production. And Cochrane Classmate is currently being piloted – a tool for building skills in evidence based practice in school and university classrooms.
TaskExchange – connecting the global evidence community to get reviews done more quickly
TaskExchange, another new platform, works as a task-matching site, matching people needing help with their SR with people who have the time and skills to help. As an open-access platform anyone can sign up and post a SR task they’d like help with, or search for tasks they’d like to complete.
Since launching in February 2016 TaskExchange has hosted almost 250 tasks and gained around 800 members, including 68 Australians.
TaskExchange meets a need identified by SR authors for extra support at various points across the lifecycle of a SR. Tasks posted on TaskExchange commonly focus on article screening and translation, data extraction and statistical support.
Task poster Melina Willson, Managing Editor, Cochrane Breast Cancer Review Group located at the NHMRC Clinical Trials Centre, University of Sydney, says
[quote]“We used Task Exchange late last year and had a quick and positive response. Four articles were translated within a week for one of our Cochrane reviews on breast reconstruction. Task Exchange is a great platform to speed up what could otherwise be a laborious process of finding people to help on a review.”[/quote]
Task responders have a variety of backgrounds including students seeking experience in evidence based practice, many of whom have formed ongoing mentor/mentee relationships with SR authors after connecting on TaskExchange.
TaskExchange can be used for any SR related task, and the platform is looking to expand into tasks related to translation of review findings into practice. Watch this space to see how this exciting platform evolves.
An important step forward
Cochrane Crowd and TaskExchange represent an important step forward in the efficient production of SRs. The platforms are two of a suite of projects conducted under the auspices of Project Transform. Together with Living Systematic Reviews and the broader framework of the Evidence Pipeline, they have the potential to make an important contribution to health evidence production.
All are welcome to sign up to Cochrane Crowd and TaskExchange. To find out more about the platforms, email firstname.lastname@example.org or email@example.com. And if you’re interested in trialling Cochrane Classmate in your classroom, please get in contact at firstname.lastname@example.org.
Conflicts of interest
Emily Steele is the Community Engagement & Partnerships Manager for TaskExchange and Cochrane Crowd. Anna Noel-Storr is co-lead on Cochrane Crowd and Information Specialist for the Cochrane Dementia and Cognitive Improvement Review Group at the University of Oxford. Tari Turner is co-lead on TaskExchange and Senior Research Fellow on Project Transform at Monash University.
Project Transform partially or fully funds all authors. Project Transform is funded as part of the Cochrane Game Changer initiative (2015-2017) and through an NHMRC Partnership Project grant (2016-2018).
Commenting on the section: “However, there are weaknesses in the SR approach. For example, SR literature searches are often more than 12 months old at the time of SR publication, and trials published in that period won’t have been included. At least 75 health care trials are published daily, so that’s a lot of available evidence not incorporated in new reviews.”
There are reasons for all this, and I don’t agree these are limitations of the SR approach.
1) FUNDS: There is a general idea that anyone can do a systematic review. This means experience in the area is not given value. Which then means no one thinks this work deserves money.
The 75 RCTs published daily are all funded, for several years and several people. With SR work, the general thought process is: we hire one who knows how to do this (often part time), and one from our existing people can do the helping bit, this should be finished within 6-9 months.
2) TIME: Anyone who has carried out a Cochrane review knows it takes longer than that to do a good review. The collaboration itself says that (http://community.cochrane.org/review-production/production-resources/proposing-and-registering-new-cochrane-reviews).
My experience says, if you spent less than 2 months on the protocol and less than a month refining searches I can guarantee you will end up with a lot of clean up needed at the end of the review.
As for updates- when you are under-resourced or working for free you can’t expect a good and timely product. Now in the third update, within past 5 years, of a Cochrane review, I can tell you it won’t have happened if the lead author wasn’t 100% committed to the task, and the guarantor had not moved many boulders to make sure the lead author got consistent funding.
3) PLANNING: SR is all about planning. I can also tell you that a poorly planned review often gets passed on at the data extraction stage to someone new and that delays things to at least another 12 months. I have recently finished two such reviews- someone planned these in a hurry and gave these to researchers inexperienced with SRs to do. When these were passed on to me and a colleague, several parts had to be redone because there were no logs of any decisions.
The only way a search can be within past 3 months of a submission is that you finalise your write up for a review in January for example, then run a search again, get two people to quickly scan/data extract/RoB assess/analyse and update the write up within the next 8 weeks and then submit to journal 1 hoping they won’t reject because you found “there is not enough good evidence”, which would take them a month to tell you (at best) and then you send to another journal and so on. Even within Cochrane groups, the time frame from a submission to publication can be between 3 to 6 months. This means a review is expected to be finished within 6- 9 months if the search is expected to be only a year old at publication. Is it not self-contradiction?
So I guess my point is, these are not limitations ‘of the SR approach’, these are consequence of ‘the approach to SRs’.
I read a long time ago somewhere that one can only do three types of jobs: a good job, a quick job, or a cheap job, but only two of these are possible together, if its good and cheap it will take time, and if its good and quick it won’t be cheap.
disclaimer: I have been carrying out systematic reviews for a decade now. Half of my contribution to date has been voluntary.