Do journalists actually read the research they’re reporting on?

Continuing the media-related theme of the previous post…

Rather than simply sending out press releases to promote new research articles and publications, it would be better if universities, research institutions and journals sent the full articles to journalists.

That is the suggestion of informal surveys of journalists that have been conducted in the UK and Australia, according to Lyndal Byford, Media Manager of the Australian Science Media Centre.

Her article below also mentions one of Croakey’s recurring gripes about online media coverage (whether of health, politics or whatever) – a widespread failure to include links to the original document so readers can go to the source themselves.

(And at the bottom of her article are links to some studies spelling out some of the pitfalls of journal press releases).

 ***

What do specialist journalists say about their reporting practices?

Lyndal Byford writes:

Science is one of those topics where reporters find themselves inundated with media releases.

In 2010, the Australian Centre for Independent Journalism analysed more than 200 health, medicine and science-related stories and found that more than half the health journalism in our papers is driven by a public relations event or media release.

Each week universities, research institutions and scientific journals swamp newsrooms across the country with release after release – but interestingly it’s rare that these releases come with an accompanying copy of the research.

A recent blog by The Guardian’s environment and science news editor James Randerson, raised an interesting question – how important is the ACTUAL scientific paper to a science journalist?

Randerson’s quick survey of his UK colleagues revealed a resounding “bloody important”, but how would Australian journalists respond?

In an effort to keep things as scientific as possible (I might want to publish a paper some day) I asked 64 specialist reporters from across Australia the same three questions Randerson had asked:

• “When writing a standard news story based on a paper in a scientific journal, how often do you get hold of the paper and read it? always/mostly/sometimes/never

• If you think it is important to read the original papers,  please explain why. How much of it do you typically read?

• If you don’t read the original paper most or all of the time, why not?”

So far I’ve received 29 responses and here’s how the Australian results break down:

When writing a standard news story based on a paper in a scientific journal, how often do you get hold of the paper and read it? Always, Always,  Always! (Always – 29, Sometimes – 0,  Never – 0)

Phew! It seems that despite the rise and rise of churnalism over journalism, specialist journalists still rely on the research papers to get to the heart of the story. Scientists across the country should breathe a sigh of relief. The paper is still mightier than the media release.

A few journalists who bravely admitted their  “always” was probably more of an “almost always” highlighted a lack of availability of the paper as the major limiting factor.

The lessons here are clear: Making papers easily available for journalists should be part and parcel of sending out the media release – to quote Sinatra – you can’t have one without the other.

But this discussion does raise another interesting question.  Since the journalists all want to read the source material – maybe the audience would too.

As more and more media outlets develop their online presence, with apps, blogs, videos and tweets galore, the science stories that provide a link to the original research are still few and far between.

Obviously not all stories lend themselves to this type of approach – feature writers in particular would suffer, having to end a story with pages of links to multiple reports.

But for your standard one study equals one story type news – the journalist’s ‘bread and butter’ as Randerson puts it – surely a link to the article only adds to the credibility of the story itself.

As pay walls go up and consumers want more from online content, links like these might go some way to being the sort of added value readers demand.

Of course that would also mean taking the research itself out from behind a pay wall – but that’s a whole other rant.

Anyway, here’s what the Australian journos had to say:

A print Journalist:

“I will read a paper cover to cover including conflicts of interest and disclosures. I never just use a media release I always ask for original statistics or a link to the original paper because if an author came out saying I had printed a completely inaccurate story I want to know I read the whole paper and was confident what I wrote was right. I think we need to be in control of the story – maybe it’s because I don’t trust anyone???”

***

Deborah Smith, Science Editor  – The Sydney Morning Herald:

“1 – Always.

2 –  To check the accuracy of the press release. The paper usually has other interesting details and quotes in it that I can use to make a better story.

It has references to other papers I can look up to work out the context and history of the research or experts to contact. I read all of it closely. The only reason I don’t read a paper is if I can’t get it in time.”

 ***

Adam Cresswell, Health Editor  – The Australian:

“Press releases often leave out essential details. And so they do, and on that logic people might assume I always put those details in my stories  –  except I don’t.

Before various people conclude I’m some sort of hypocrite, I should add that I need to know these details partly to assess the newsworthiness of a story, but there might well not be space to include them.

For example, the fact that a JAMA study a couple of years ago about Viagra’s effect on female sexual dysfunction was funded by Pfizer (a fact hidden at the end of the full paper) played some considerable part in my decision not to cover the story. But there have been other cases where info, for example about the methodology, or the funding source, gives added confidence that the findings are important, but do not themselves get a mention because there are too many more important details to fit in to the limited space.

I would normally try to ensure I did include the fact a study had been funded by a drug’s maker, since that is a clear potential conflict. But the fact a study had been funded by the US National Institutes of Health – while very useful to know  –  would be far less likely to be included, given space constraints.

Likewise, I know Media Doctor and others seem to think every story about a study should say whether a study is an RCT, cohort study or whatever, but they seem blissfully unconcerned that most readers simply do not understand what these terms mean, and it chews up a good paragraph or two of the story to educate them. Often, that just squeezes out other details that ultimately are more important and more interesting for the general (as opposed to medical or science professional) readership.

Ultimately, it is up to the journalist writing the story to consider all the relevant facts about a story, and report the study – if asked to do so, in a very short space indeed  – in a way that puts the findings in the appropriate context (including attaching suitable cautions if the methodology is weak, numbers small, etc). They very rarely have space to repeat every salient fact in the study itself, and they shouldn’t have to, as the study is its own witness to all that. If the journalist ends up getting the study out of proportion, eg effectively writing a beat-up by relying on relative risk only without reference to absolute risk, then that will affect their credibility in the longer term.”

 ***

A freelance journalist:

“A lot of the details that you need in order to tell the story properly are easiest to get directly from the paper itself. It’s possible to ask the researchers themselves, but often the paper actually provides prompts for questions you might not otherwise have considered. I read the abstract first, then the introduction and the discussion/conclusions then the results, and methods and results.”

 ***

Leigh Dayton, Science Writer, Editor Weekend Health – The Australian:

“It’s critical to read the paper to find out, well, what it says. There may be a misleading institution/journal-spin in the press release or a very important element might be overlooked.  I usually read the entire paper unless it’s jammed with mathematical analysis that’s over my head, in which case I read as much of it as I can and send it out for an opinion to an expert to confirm or otherwise it’s significance.

A specialist who doesn’t read the paper is lazy, irresponsible and a very naughty person. That’s especially worrisome as there’s a decline in specialist science writers in particular.”

 ***

Darren Osborne, News Editor  –  ABC Science online:

“As news editor there are a number of reasons why I check the scientific paper a story is based on. Firstly, to check the numbers. There have been occasions when a journalist has put in the wrong figures (whether through fault of their own or others). Next, I like to check some of the figures, in particular percentages and ratios and compare them with the ‘raw’ numbers. This is useful when translating relative risk to absolute risk.

Another reason – and this applies more to medical research – is to search for conflicts of interest and sources of funding.

Finally, it’s good to sometimes good to see the detail – methodology, discussion, etc.

I should point out that I don’t read every paper, most times it’s a skim read. But all are downloaded. We also link to the abstract in the stories  ( hooray – Lyndal) , using the DOI number.”

 ***

Jane Hammond, Journalist/Science Writer  –  The West Australian:

“Important to get my own take on the paper and to be sure the abstract is sufficient and correct, also to get additional background and to go beyond a press release. I read the bulk of the paper skipping over the mathematical bits and overly scientific explanations, unless really necessary.”

 ***

Julian Drape, Climate Change and Health reporter – AAP:

“I think the full report is essential even if I only look at the exec summary and then go to selected sections to follow up in more detail. I would never write a story based off a press release alone. The PR people may have got things wrong for starters!”

 ***

A print journalist:

“I typically read the entire paper. The most interesting part of the story (or the story itself) is not always contained in the media release. And I like to check the validity of what is asserted in the media release. There can also be interesting graphics or statistics we might want to reproduce and references to other studies.

On a very small number of occasions I have not read the full paper because it hasn’t been available but I am extremely reluctant to do so.”

***

Health Reporter – ABC:

“I always try and read the full paper when I can. The full paper gives you more details and analysis. I read the whole paper but especially  the discussion and conclusions”

 ***

Broadcaster and journalist – ABC:

“I always use the scientific paper as the basis for any new research story I write. The presser is only ever a broad guide.

I read the entire paper – I don’t always understand it, so where possible I contact either the researcher who wrote it or get a local scientist to comment.”

*** 

Conor Duffy, Environment Reporter  – ABC:

“I always have a look at the source paper. I would like access to this all the time. I think it’s important to satisfy yourself about the quality of the research. At times it will be difficult to read an entire paper (where it runs to hundreds or sometimes even thousands of papers). In those cases I would still like to see the paper to check the context of the parts of the paper I intend on quoting from.”

 ***

Stephen Cauchi, Technology Reporter  – The Sunday Age

When writing a standard news story based on a paper in a scientific journal how often do you get hold of the paper and read it? always/mostly/sometimes/never — ALWAYS

If you think it is important to read the original paper please explain why? How much of it do you typically read? — BETTER UNDERSTANDING.

I TRY TO READ AS MUCH AS POSSIBLE BUT TEND TO SKIP THE OVERLY TECHNICAL STUFF.

 ***

Kate Hagan, Health Reporter – The Age:

“I always get hold of the paper and read it. I think it is important for journalists to look at source documents and not rely on a media release (and therefore someone else’s interpretation of what is important). There are rare papers that are extremely technical and difficult for a layperson to read but I always have a go. It is also of course very important to canvas experts in the area.”

 ***

Nicky Phillips, Science and Technology reporter – The Sydney Morning Herald:

“Always. I wouldn’t even write a 150 word brief based on a scientific article without reading it first. When we run wire copy based on a paper I will almost always read the paper and cross check them.

As part of being a journalist it is important that I try and get as close to the source material as possible. I can’t trust what is written in a press release for several reasons: university PR have their own agenda when writing press releases and the parts they find most compelling are not necessarily the same parts of the research our readers would find most compelling. Accuracy is also always an issue in press releases, as is the interpretation of the results. Plus, most of the great detail is buried in the journal article. For me, it’s a must read.”

 ***

Thomas Arup, Environment Reporter – The Age:

“Always. As a reporter I need the best understanding possible of the issue I am writing about. It is important to get across caveats which aren’t often in a press release. Press releases are also normally someone else’s interpretation of the work and it is better I get my own full understanding. I normally seek to speak with one of the main researchers too. I normally read the complete paper, but will skip over heavy maths sections etc. Occasionally when deadlines are tight I will rely on an executive summary.”

• Thanks to the Australian Science Media Centre for allowing cross-posting of this article

PS from Croakey

Professors Lisa Schwartz and Steven Woloshin, from Dartmouth Medical School in the US, have been at the forefront of efforts to study the relationship between media reporting and the quality of journal abstracts, articles and press releases, as per these studies:

• Medical journal press releases are of variable quality, and the better ones are associated with better quality news stories.

• Press releases from academic medical centers often promote research that has uncertain relevance to human health and do not provide key facts or acknowledge important limitations.

• Press releases do not routinely highlight study limitations or the role of industry funding. Data are often presented using formats that may exaggerate the perceived importance of findings.

***

Update, 4 April

A call to rethink medical conferences

On related themes, US health journalism watchdog Gary Schwitzer has urged journalists to rethink their approaches to reporting on medical conferences. Coverage of preliminary findings from conferences can mislead media audiences, he says. 

Schwitzer cites a JAMA paper, Are Medical Conferences Useful? And for Whom?, by John Ioannidis, from the Stanford Prevention Research Center in California, which says there is very little evidence that medical conferences achieve their goal of disseminating and advancing research, training, education, and evidence-based policy.

On the other hand, Ioannidis says there is some evidence suggesting that medical congresses may serve a specific system of questionable values that may be harmful to medicine and health care – from fuel wastage to enabling industry influence.

He writes: “In the electronic age in which information can be shared around the world instantly, the contribution of large medical conferences to the dissemination and advancement of science is unclear.”

Ioannidis suggests some experiments, including:

• Setting more stringent criteria for selecting who organises medical congresses. For example, one option is to exclude from the organisation committees (and also from the leadership of professional medical societies) all investigators with any ties to the industry in the last 3 years.

• Formal studies to assess what types of meetings or other methods for research dissemination and education work best in training excellent physicians, improving medical care, and controlling cost.

There are some advantages to health and medical conferences from a journalistic viewpoint, however. At least they can help get us away from our desks and computers, and out where we have more of a chance of being exposed (hopefully) to some real world stories and conversations. Unfortunately the reality of constrained newsrooms is that there is a heavy reliance upon conference PR machines.

(The JAMA abstract is here. Please leave your contact details below or email me direct if you’d like a copy of the full paper).