Informed, engaged communities for health

Filter by Categories
#PHAAThinkTank2023
Aged care
Budgets
Federal Budget 2023-2024
Federal Budget October 2022
Federal Budget 2022-23
Federal Budget 2021-22
Budget2020Health
Federal Budget 2020-21
Federal Budget 2019-20
Climate and health
National Health and Climate Strategy
#HealthyCOP28
Climate emergency
Heatwaves
Healthcare sustainability
#CoveringClimateNow
#HealthyCOP27
#HealthyCOP26
COVID collection
Long COVID
COVID-19
COVIDwrap
COVID SNAPS
#JusticeCOVID
Caring for the Frontline
COVIDglobalMHseries
Croakey Conference News Service
#NSPC23
Our Democracy Forum
#AskMSF
#Lowitja2023
#GreenHealthForum23
#hpsymposium2023
#NMS23
#HEAL2023
#ASMIRT2023
#16nrhc
#GreenHealthForum22
#Heal2022
#ICEM22
#NAISA22
#NNF2022
#RANZCP2022
#RethinkAddiction
#RTP22
Choosing Wisely National Meeting 2022
Equally Well 2022 Symposium
GiantSteps22
Croakey Professional Services
NHLF series
#KidneyCareTogether
ACSQHC series
ACSQHC series 2022
ACSQHC series 2021
ACSQHC series 2020
ACSQHC series 2019
CATSINaM 25 Years
Croakey projects
#CroakeyLIVE #DigitalNationBuilding
#CroakeyLIVE #VoiceForHealth
The Health Wrap
ICYMI
@WePublicHealth
@WePublicHealth2023
#CroakeyVOICES
#SpeakingOurMinds
Croakey longreads
#CroakeyREAD
CroakeyEXPLORE
#PHAAThinkTank 2022
Summer reading 2022-2023
CroakeyGO
#CroakeyGO #NavigatingHealth
#GamblingHarms
#HeatwaveHealth
Mapping CroakeyGo
Determinants of health
Environmental determinants of health
Social determinants of health
Discrimination
Racism
Justice and policing
Poverty
Newstart/JobSeeker
Education
Housing
Internet access
Justice Reinvestment
Social policy
Commercial determinants of health
fossil fuels
Alcohol
Digital platforms
Food and beverages
Sugar tax
Tobacco
Vaping
Plain packaging
Gambling
Pharmaceutical industry
Arms industry
Disasters and extreme weather events
Disasters
Extreme weather events
Bushfires
Bushfire-emergency 2019-2020
Floods 2023
Floods 2022
Floods 2021
Floods 2011
Donor-funded journalism
Donor-funded journalism – 2023
Donor-funded journalism – 2022
Donor-funded journalism – 2021
Donor-funded journalism – 2020
Elections
#NSWvotesHealth2023
Victorian election 2022
Federal Election 2022
The Election Wrap 2022
#QldVotesHealth
SA election 2022
WA election 2021
Tasmanian election 2021
First Nations
Indigenous health
Uluru Statement
The Voice
Community controlled sector
Cultural determinants of health
Cultural safety
Social and emotional wellbeing
Indigenous education
Lowitja Institute
NT Intervention
WA community closures
Acknowledgement
#CTG10
#NTRC
#RCIADIC30Years
General health matters
Consumer health matters
Cancer
Cardiovascular disease
Chronic conditions
Non communicable diseases
Diabetes
Oral health
Disabilities
Fetal Alcohol Spectrum Disorders (FASD)
Genetics
Medical marijuana
Mental health
Suicide
Trauma
LGBTQIA+
HIV/AIDS
Women's health
HRT
Abortion
Men's health
Youth health
Child health
Pregnancy and childbirth
Organ transplants
Pain
Sexual health
Infectious diseases
Influenza
Mpox
Swine flu
Death and dying
Euthanasia
Global health matters
Global health
WHO
Conflict and war
Asylum seeker and refugee health
#WorldInTurmoil
Ebola
NHS
Health policy and systems
MyMedicare
Health reform
Workforce matters
Health financing and costs
Health regulation
Strengthening Medicare Taskforce 2022
Co-design
TGA
Royal Commissions
National Commission of Audit 2014
National Health Performance Authority
Pharmaceutical Benefits Scheme
Private health insurance
Healthcare
Primary healthcare
cohealth
Social prescribing
Rural and remote health
General practice
International medical graduates
Primary Health Networks
NDIS
Nursing and midwifery
Australian Medical Association
Allied healthcare
Pharmacy
Paramedics
Pathology
Naturopathy
Hospitals
Emergency departments and care
Palliative care
Safety and quality of healthcare
Adverse events
Choosing Wisely
Co-payments
Out of pocket costs
Complementary medicines
Conflicts of interest
Health ethics
Digital technology
Telehealth
E-health
Equally Well
Health Care Homes
Medicare Locals
MyHospitals website
Screening
Surgery
Tests
Media and health
Media-related issues
Public interest journalism
Misinformation and disinformation
Social media and healthcare
Health & medical marketing
The Conversation
Media Doctor Australia
News about Croakey
Public health and population health
Artificial intelligence
Public health
Prevention
Health inequalities
Human rights
Health in All Policies
Health impact assessment
Legal issues
Australian Centre for Disease Control
VicHealth
Health literacy
Health communications
Vaccination
Air pollution
Physical activity
Sport
Obesity
Road safety
Transport
Gun control
Illicit drugs
Injuries
Marriage equality
Occupational health
Violence
Weight loss products
#PreventiveHealthStrategy
#UnmetNeedsinPublicHealth
Government 2.0
Web 2.0
Nanny state
National Preventive Health Agency
Research matters
Health and medical research
#MRFFtransparency
Evidence-based issues
Cochrane Collaboration
Health and medical education
NHMRC
The Croakey Archives
#cripcroakey
#HealthEquity16
#HealthMatters
#IHMayDay (all years)
#IHMayDay 2014
#IHMayDay15
#IHMayday16
#IHMayDay17
#IHMayDay18
#LoveRural 2014
Croakey Conference News Service 2013 – 2021
2021 conferences
#21OPCC
#BackToTheFire
#FoodGovernance2021
#GiantSteps21
#GreenHealthForum21
#HealthClimateSolutions21
#HearMe21
#IndigenousClimateJustice21
#NNF2021
#RANZCP2021
#ShiftingGearsSummit
#ValueBasedCare
#WCepi2021
#YHFSummit
2020 conferences
#2020ResearchExcellence
#Govern4Health
#HealthReImagined
#SAHeapsUnfair
2019 Conferences
#ACEM19
#CPHCE19
#EquallyWellAust
#GiantSteps19
#HealthAdvocacyWIM
#KTthatWorks
#LowitjaConf2019
#MHAgeing
#NNF2019
#OKtoAsk2019
#RANZCOG19
#RANZCP2019
#ruralhealthconf
#VMIAC2019
#WHOcollabAHPRA
2018 conferences
#6rrhss
#ACEM18
#AHPA2018
#ATSISPC18
#CPHCE
#MHED18
#NDISMentalHealth
#Nurseforce
#OKToAsk2018
#RANZCOG18
#ResearchIntoPolicy
#VHAawards
#VMIACAwards18
#WISPC18
2017 conferences
#17APCC
#ACEM17
#AIDAconf2017
#BTH20
#CATSINaM17
#ClimateHealthStrategy
#IAHAConf17
#IDS17
#LBQWHC17
#LivingOurWay
#OKtoAskAu
#OTCC2017
#ResearchTranslation17
#TheMHS2017
#VMIACConf17
#WCPH2017
Australian Palliative Care Conference
2016 conferences
#AHHAsim16
#AHMRC16
#ANROWS2016
#ATSISPEP
#AusCanIndigenousWellness
#cphce2016
#CPHCEforum16
#CRANAplus2016
#IAMRA2016
#LowitjaConf2016
#PreventObesity16
#TowardsRecovery
#VMIAC16
#WearablesCEH
#WICC2016
2015 conferences
#CPHCEforum
#CRANAplus15
#HSR15
#NRHC15
#OTCC15
Population Health Congress 2015
2014 conferences
#IPCHIV14
AIDA Conference 2014
Congress Lowitja 2014
CRANAplus conference 2014
Cultural Solutions - Healing Foundation forum 2014
Lowitja Institute Continuous Quality Improvement conference 2014
National Suicide Prevention Conference 2014
Racism and children/youth health symposium 2014
Rural & Remote Health Scientific Symposium 2014
2013 conferences
Australian Centre for Health Services Innovation Forum 2013
Australian Health Promotion Association Conference 2013
Closing the Credibility Gap 2013
CRANAplus Conference 2013
FASD Conference 2013
Health Workforce Australia 2013
International Health Literacy Network Conference 2013
NACCHO Summit 2013
National Rural Health Conference 2013
Oceania EcoHealth Symposium 2013
PHAA conference 2013
Croakey Professional Services archive
#CommunityControl
#CommunityControl Twitter Festival
#COVIDthinktank21
Lowitja Indigenous knowledge translation series
Croakey projects archive
#CommunityMatters
#CroakeyFundingDrive 2022
#CroakeyLIVE #Budget2021Health
#CroakeyLIVE #USvotesHealth
#CroakeyLIVE Federal election 2022
#CroakeyYOUTH
#HousingJusticeAus
#IndigenousHealthSummit
#IndigenousNCDs
#JustClimate
#JustJustice
#LookingLocal
#OutOfPocket
#OutOfTheBox
#RuralHealthJustice
#TalkingTeeth
@WePublicHealth2022
@WePublicHealth2021
@WePublicHealth2020
AroundTheTraps
Croakey register of influence
Croakey Register of Influencers in Public Health
Croakey Register of Unreleased Documents
Gavin Mooney
Inside Story
Journal Watch
Naked Doctor
Poems of Public Health
Summer reading 2021-2022
Summer reading 2020-2021
Summer Reading 2019-2020
Summer Reading 2017-2018
Summer Reading 2016-2017
The Koori Woman
TOO MUCH of a Good Thing
Wonky Health
CroakeyGO archive 2017 – 2018
CroakeyGo 2018
#CroakeyGO #QuantumWords 2018
#CroakeyGO #VicVotes 2018
#CroakeyGO Albury 2018
#CroakeyGO Callan Park 2018
#CroakeyGO Carnarvon 2018
#CroakeyGO Marrickville 2018
#CroakeyGO Palm Island 2018
CroakeyGo 2017
#CroakeyGO Adelaide 2017
#CroakeyGO Melbourne 2017
#CroakeyGO Newcastle 2017
#CroakeyGO Sydney 2017
Elections and Budgets 2013 – 2019
#AusVotesHealth Twitter Festival 2019
#Health4NSW
Federal Election 2019
NSW Election 2019
Federal Budget 2018-19
Federal Budget 2017/18
NZ Election 2017
Federal Budget 2016-17
Federal Election 2016
#HealthElection16
NT Election 2016
Federal Budget 2015-16
Qld Election 2015
NSW Election 2015
Federal Budget 2014-15
Victorian Election 2014
Federal Budget 2013-14
Federal Election 2013
Federal Budget 2012-2013
Federal Budget 2011
Federal Budget 2010
Federal Election 2010
Federal Budget 2009-2010
Support non-profit public interest journalism
Filter by Categories
#PHAAThinkTank2023
Aged care
Budgets
Federal Budget 2023-2024
Federal Budget October 2022
Federal Budget 2022-23
Federal Budget 2021-22
Budget2020Health
Federal Budget 2020-21
Federal Budget 2019-20
Climate and health
National Health and Climate Strategy
#HealthyCOP28
Climate emergency
Heatwaves
Healthcare sustainability
#CoveringClimateNow
#HealthyCOP27
#HealthyCOP26
COVID collection
Long COVID
COVID-19
COVIDwrap
COVID SNAPS
#JusticeCOVID
Caring for the Frontline
COVIDglobalMHseries
Croakey Conference News Service
#NSPC23
Our Democracy Forum
#AskMSF
#Lowitja2023
#GreenHealthForum23
#hpsymposium2023
#NMS23
#HEAL2023
#ASMIRT2023
#16nrhc
#GreenHealthForum22
#Heal2022
#ICEM22
#NAISA22
#NNF2022
#RANZCP2022
#RethinkAddiction
#RTP22
Choosing Wisely National Meeting 2022
Equally Well 2022 Symposium
GiantSteps22
Croakey Professional Services
NHLF series
#KidneyCareTogether
ACSQHC series
ACSQHC series 2022
ACSQHC series 2021
ACSQHC series 2020
ACSQHC series 2019
CATSINaM 25 Years
Croakey projects
#CroakeyLIVE #DigitalNationBuilding
#CroakeyLIVE #VoiceForHealth
The Health Wrap
ICYMI
@WePublicHealth
@WePublicHealth2023
#CroakeyVOICES
#SpeakingOurMinds
Croakey longreads
#CroakeyREAD
CroakeyEXPLORE
#PHAAThinkTank 2022
Summer reading 2022-2023
CroakeyGO
#CroakeyGO #NavigatingHealth
#GamblingHarms
#HeatwaveHealth
Mapping CroakeyGo
Determinants of health
Environmental determinants of health
Social determinants of health
Discrimination
Racism
Justice and policing
Poverty
Newstart/JobSeeker
Education
Housing
Internet access
Justice Reinvestment
Social policy
Commercial determinants of health
fossil fuels
Alcohol
Digital platforms
Food and beverages
Sugar tax
Tobacco
Vaping
Plain packaging
Gambling
Pharmaceutical industry
Arms industry
Disasters and extreme weather events
Disasters
Extreme weather events
Bushfires
Bushfire-emergency 2019-2020
Floods 2023
Floods 2022
Floods 2021
Floods 2011
Donor-funded journalism
Donor-funded journalism – 2023
Donor-funded journalism – 2022
Donor-funded journalism – 2021
Donor-funded journalism – 2020
Elections
#NSWvotesHealth2023
Victorian election 2022
Federal Election 2022
The Election Wrap 2022
#QldVotesHealth
SA election 2022
WA election 2021
Tasmanian election 2021
First Nations
Indigenous health
Uluru Statement
The Voice
Community controlled sector
Cultural determinants of health
Cultural safety
Social and emotional wellbeing
Indigenous education
Lowitja Institute
NT Intervention
WA community closures
Acknowledgement
#CTG10
#NTRC
#RCIADIC30Years
General health matters
Consumer health matters
Cancer
Cardiovascular disease
Chronic conditions
Non communicable diseases
Diabetes
Oral health
Disabilities
Fetal Alcohol Spectrum Disorders (FASD)
Genetics
Medical marijuana
Mental health
Suicide
Trauma
LGBTQIA+
HIV/AIDS
Women's health
HRT
Abortion
Men's health
Youth health
Child health
Pregnancy and childbirth
Organ transplants
Pain
Sexual health
Infectious diseases
Influenza
Mpox
Swine flu
Death and dying
Euthanasia
Global health matters
Global health
WHO
Conflict and war
Asylum seeker and refugee health
#WorldInTurmoil
Ebola
NHS
Health policy and systems
MyMedicare
Health reform
Workforce matters
Health financing and costs
Health regulation
Strengthening Medicare Taskforce 2022
Co-design
TGA
Royal Commissions
National Commission of Audit 2014
National Health Performance Authority
Pharmaceutical Benefits Scheme
Private health insurance
Healthcare
Primary healthcare
cohealth
Social prescribing
Rural and remote health
General practice
International medical graduates
Primary Health Networks
NDIS
Nursing and midwifery
Australian Medical Association
Allied healthcare
Pharmacy
Paramedics
Pathology
Naturopathy
Hospitals
Emergency departments and care
Palliative care
Safety and quality of healthcare
Adverse events
Choosing Wisely
Co-payments
Out of pocket costs
Complementary medicines
Conflicts of interest
Health ethics
Digital technology
Telehealth
E-health
Equally Well
Health Care Homes
Medicare Locals
MyHospitals website
Screening
Surgery
Tests
Media and health
Media-related issues
Public interest journalism
Misinformation and disinformation
Social media and healthcare
Health & medical marketing
The Conversation
Media Doctor Australia
News about Croakey
Public health and population health
Artificial intelligence
Public health
Prevention
Health inequalities
Human rights
Health in All Policies
Health impact assessment
Legal issues
Australian Centre for Disease Control
VicHealth
Health literacy
Health communications
Vaccination
Air pollution
Physical activity
Sport
Obesity
Road safety
Transport
Gun control
Illicit drugs
Injuries
Marriage equality
Occupational health
Violence
Weight loss products
#PreventiveHealthStrategy
#UnmetNeedsinPublicHealth
Government 2.0
Web 2.0
Nanny state
National Preventive Health Agency
Research matters
Health and medical research
#MRFFtransparency
Evidence-based issues
Cochrane Collaboration
Health and medical education
NHMRC
The Croakey Archives
#cripcroakey
#HealthEquity16
#HealthMatters
#IHMayDay (all years)
#IHMayDay 2014
#IHMayDay15
#IHMayday16
#IHMayDay17
#IHMayDay18
#LoveRural 2014
Croakey Conference News Service 2013 – 2021
2021 conferences
#21OPCC
#BackToTheFire
#FoodGovernance2021
#GiantSteps21
#GreenHealthForum21
#HealthClimateSolutions21
#HearMe21
#IndigenousClimateJustice21
#NNF2021
#RANZCP2021
#ShiftingGearsSummit
#ValueBasedCare
#WCepi2021
#YHFSummit
2020 conferences
#2020ResearchExcellence
#Govern4Health
#HealthReImagined
#SAHeapsUnfair
2019 Conferences
#ACEM19
#CPHCE19
#EquallyWellAust
#GiantSteps19
#HealthAdvocacyWIM
#KTthatWorks
#LowitjaConf2019
#MHAgeing
#NNF2019
#OKtoAsk2019
#RANZCOG19
#RANZCP2019
#ruralhealthconf
#VMIAC2019
#WHOcollabAHPRA
2018 conferences
#6rrhss
#ACEM18
#AHPA2018
#ATSISPC18
#CPHCE
#MHED18
#NDISMentalHealth
#Nurseforce
#OKToAsk2018
#RANZCOG18
#ResearchIntoPolicy
#VHAawards
#VMIACAwards18
#WISPC18
2017 conferences
#17APCC
#ACEM17
#AIDAconf2017
#BTH20
#CATSINaM17
#ClimateHealthStrategy
#IAHAConf17
#IDS17
#LBQWHC17
#LivingOurWay
#OKtoAskAu
#OTCC2017
#ResearchTranslation17
#TheMHS2017
#VMIACConf17
#WCPH2017
Australian Palliative Care Conference
2016 conferences
#AHHAsim16
#AHMRC16
#ANROWS2016
#ATSISPEP
#AusCanIndigenousWellness
#cphce2016
#CPHCEforum16
#CRANAplus2016
#IAMRA2016
#LowitjaConf2016
#PreventObesity16
#TowardsRecovery
#VMIAC16
#WearablesCEH
#WICC2016
2015 conferences
#CPHCEforum
#CRANAplus15
#HSR15
#NRHC15
#OTCC15
Population Health Congress 2015
2014 conferences
#IPCHIV14
AIDA Conference 2014
Congress Lowitja 2014
CRANAplus conference 2014
Cultural Solutions - Healing Foundation forum 2014
Lowitja Institute Continuous Quality Improvement conference 2014
National Suicide Prevention Conference 2014
Racism and children/youth health symposium 2014
Rural & Remote Health Scientific Symposium 2014
2013 conferences
Australian Centre for Health Services Innovation Forum 2013
Australian Health Promotion Association Conference 2013
Closing the Credibility Gap 2013
CRANAplus Conference 2013
FASD Conference 2013
Health Workforce Australia 2013
International Health Literacy Network Conference 2013
NACCHO Summit 2013
National Rural Health Conference 2013
Oceania EcoHealth Symposium 2013
PHAA conference 2013
Croakey Professional Services archive
#CommunityControl
#CommunityControl Twitter Festival
#COVIDthinktank21
Lowitja Indigenous knowledge translation series
Croakey projects archive
#CommunityMatters
#CroakeyFundingDrive 2022
#CroakeyLIVE #Budget2021Health
#CroakeyLIVE #USvotesHealth
#CroakeyLIVE Federal election 2022
#CroakeyYOUTH
#HousingJusticeAus
#IndigenousHealthSummit
#IndigenousNCDs
#JustClimate
#JustJustice
#LookingLocal
#OutOfPocket
#OutOfTheBox
#RuralHealthJustice
#TalkingTeeth
@WePublicHealth2022
@WePublicHealth2021
@WePublicHealth2020
AroundTheTraps
Croakey register of influence
Croakey Register of Influencers in Public Health
Croakey Register of Unreleased Documents
Gavin Mooney
Inside Story
Journal Watch
Naked Doctor
Poems of Public Health
Summer reading 2021-2022
Summer reading 2020-2021
Summer Reading 2019-2020
Summer Reading 2017-2018
Summer Reading 2016-2017
The Koori Woman
TOO MUCH of a Good Thing
Wonky Health
CroakeyGO archive 2017 – 2018
CroakeyGo 2018
#CroakeyGO #QuantumWords 2018
#CroakeyGO #VicVotes 2018
#CroakeyGO Albury 2018
#CroakeyGO Callan Park 2018
#CroakeyGO Carnarvon 2018
#CroakeyGO Marrickville 2018
#CroakeyGO Palm Island 2018
CroakeyGo 2017
#CroakeyGO Adelaide 2017
#CroakeyGO Melbourne 2017
#CroakeyGO Newcastle 2017
#CroakeyGO Sydney 2017
Elections and Budgets 2013 – 2019
#AusVotesHealth Twitter Festival 2019
#Health4NSW
Federal Election 2019
NSW Election 2019
Federal Budget 2018-19
Federal Budget 2017/18
NZ Election 2017
Federal Budget 2016-17
Federal Election 2016
#HealthElection16
NT Election 2016
Federal Budget 2015-16
Qld Election 2015
NSW Election 2015
Federal Budget 2014-15
Victorian Election 2014
Federal Budget 2013-14
Federal Election 2013
Federal Budget 2012-2013
Federal Budget 2011
Federal Budget 2010
Federal Election 2010
Federal Budget 2009-2010

Beyond the p(ale) — statistics and significance in clinical research

With a boom in predatory journals and “publish or perish” pressures more real than ever, truly significant findings are becoming ever-more difficult to discern, prompting proposals for a rethink on how we define new discoveries.

In an upcoming piece for Nature Human Behaviour, published on preprint this month, a global transdisciplinary group of researchers have called for the threshold on a common benchmark of significance — the p-value — to be dropped from 0.05 to 0.005, arguing that the former allows too many false positives, even in the most robust of studies.

Put simply, in research the p-value is an expression of how likely it is that your results are due to chance. Traditionally, p=<0.05 is said to be statistically significant, putting this likelihood at less than 5%, though some authors prefer p=<0.01 (less than 1%) as a threshold.

The Nature authors propose tossing these markers in favour of p=<0.005 (less than 0.5%) as a new benchmark in the social and biomedical sciences.

But will this address what’s been described as a “reproducibility crisis” in science?

ben1

ben2ben3ben4Shlomo Argamon, Professor of Computer Science and Director of the Master of Data Science at the Illinois Institute of Technology, argues that it may in fact entrench woes within the current system, and has called for a rethink on how we approach the publication of research, in a series of blog posts republished with permission for Croakey readers.

(And if stats and bias is your thing, you might be interested in a recent proposal in the Annals of Internal Medicine introducing an ‘e-value’ to measure causality and confounding factors in observational studies).


Shlomo Argamon writes:

Several days ago, an article was posted on PsyArXiv (to appear in Nature Human Behavior), arguing that a significant step towards solving the reproducibility crisis in many fields will be to reduce the standard significance threshold to require p < 0.005 (rather than the current standard of 0.05) for a result to be termed “statistically significant”. Results with higher p values could only be termed “suggestive” under this proposal.

I, along with a number of other scientists, was recently interviewed for a brief article in Nature | News which clearly, and concisely, sets out arguments for and against this proposal as a way to address the reproducibility crisis. I’d like to expand here on my very compressed comments as quoted in that article.

On the one hand, the proposal to reduce the “significance threshold” to 0.005 will indeed be helpful, in that the change will reduce the occurrence of false positives, and thus enhance the likelihood that effects termed “significant” will be reproducible, and so reduce the number of non-existent effects reported in the literature as real.

However, the proposal does nothing whatsoever to change the fundamental problem, which is the entire framework of judging experimental results by a binary criterion of “significance”. There are two key problems with the regnant view of “statistical significance”:

  1. It does not account for effect size – a large effect with less-significant p may have much more “real-life significance” than a tiny effect with more-significant p, and
  2. It is very easy to (even inadvertently) “p-hack” and find putatively significant effects due to the very large number of possible parameter settings for any experiment in a sufficiently rich and theoretically-underdetermined domain.

Given that, this new recommendation may even do significant harm, in that improving research outcomes somewhat (by lowering false positive rates) without addressing the fundamental problem could lead to a false sense of security and entrench bad methodological attitudes.

Same old song

To expand on this, the problem I have with the new recommendation is that it maintains the exact same conceptual framework of statistical significance testing that has been a major driver of the replicability crisis.

In statistical significance testing, a level of confidence (previously 0.05, per the new proposal 0.005) is defined, and then the question asked is: Is the probability that the null hypothesis produced the experimental results less than that confidence level? If it is, the null hypothesis is rejected, and the hypothesis is considered statistically significant.

This framework does not account at all for the question of effect size – an effect may be statistically significant, but practically speaking negligible and irrelevant.

For example, often the null hypothesis is of the form “x = 0” for some model parameter x. Any such null hypothesis is almost certainly false, since the actual value of x is probably at least slightly different from 0, and so some statistically significant result can nearly always be extracted with enough data and ingenuity, even though the result may not be significant in any meaningful way in the real world.

Forking paths and the file drawer effect

More fundamentally, no matter what confidence level you choose, if there are enough different ways to design your experiment, it becomes highly likely that at least one of them will give a statistically significant result just by chance. And unless every single detail of the experiment is decided and registered ahead of time, it is virtually impossible to ensure, except by care and good judgement by the researchers, that no choices are made along the way that could (unwittingly) enhance the significance of the final experimental results, since there are so many different choices.

This phenomenon was termed the “garden of forking paths” by Gelman and Loken, and illustrated nicely by XKCD. (This problem is inherent in any metric-based decision criterion for identifying effects, and is not specific to statistical significance testing.)

Finally, the goal and the strength of the “p < 0.005” proposal is that it will reduce the number of false positive results in the literature.

However, by the same token, there will be fewer reports of “less statistically significant” results, which will cause overall problems for the science, due to the “file-drawer” effect.

The problem is that the vast majority of negative results are never reported, which gives a biased picture of the state of knowledge. By lowering the p-threshold, the file-drawer bias will be even greater, since fewer results, with only lower p-values, are published.

So how should we proceed?

There is no simple answer to this question, as there are several interrelated problems: p-hacking, effect sizes, and literature bias, as I’ve mentioned above, but also the question of how the incentive structures of modern science* may bias the creation and evaluation of new knowledge (the file-drawer problem is an example of this).

All of these factors contribute to the “reproducibility crisis”, and all need to be addressed. Any simple answer to one of these, such as changing the threshold p-value, will not solve the problem, and may in fact make things worse by making it seem solved when it is not.

Any meaningful solution must involve a combination of new methodological standards, institutional structures, and incentives.

Reducing the p threshold is simple and could be one useful methodological standard, though it would perpetuate problematic concepts, as I’ve said. Another would be introducing some notions from Bayesian statistics, such as

  1. Using Bayes factors, which are more easily interpreted in most situations than a p-value, as a standard reported metric, or
  2. The use of explicit priors, to make assumptions clearer, or
  3. Explicit loss functions, to account explicitly for the importance of effect size and the varying weights of different kinds of errors.

It is essential that the scientific community have good incentives for publishing replications, especially those that fail, and for publishing negative results.  (And there cannot be penalties for early-stage researchers for doing so!)

There need to be more and better institutional structures for preregistering studies, like clinicaltrials.gov, but set up with the appropriate community standards and consequences, so that they actually work.

Just as during the early and laborious development of effective scientific practices and institutions several hundred years ago, there is no royal road to solving the crisis.

A great deal of hard work lies ahead of us, but past experience has shown that it will (eventually) pay off in more solid understanding of our world.

*See a follow-up post on perverse incentives of the new proposal here

Shlomo Argamon is a Professor of Computer Science and Director of the Master of Data Science at the Illinois Institute of Technology

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search by: Categories or tags