A holistic approach to empirical analysis: The insignificance of P, hypothesis testing and statistical significance

Morris Altman (Lead / Corresponding author)

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

It is well documented that academics and practitioners focus on statistical significance (typically represented by P tests) and statistical hypothesis testing to determine if their non-statistical analytical hypothesis is correct or likely to be correct. Moreover, statistical significance is relied upon to determine which variables should be used in their models or analyses. In spite of ongoing criticism, this practice continues to the detriment of robust scientific analysis. I discuss the significant limitations of statistical significance in scientific analysis, irrespective of discipline, with some focus on economics. I place statistical significance in a broader analytical context, discussing other analytical procedures that need to be followed and emphasized for one’s analysis to be scientifically robust. This relates the development of models, assumptions underlying the models, the data collected and constructed, the relationship between statistical significance and causality and the importance of non-statistical theory to the identification of pertinent modelling variables. Analytical significance (size effects and variability) is core to any robust scientific analysis, but only in the context of all of the other prior steps in the applied research project being in place. In this broader analytical framework, the statistical significance becomes relatively insignificant. I also address why statistical significance and statistical hypothesis testing dominates the applied analytical landscape even though this dominance is not best practice. Of critical importance are the mental models of best practice and the worldview and preferences of decision makers determining what gets published and who is successful in securing grants and employment.

Original languageEnglish
Title of host publicationFrom Analysis to Visualization
Subtitle of host publicationA Celebration of the Life and Legacy of Jonathan M. Borwein, Callaghan, Australia, September 2017
EditorsDavid H. Bailey, Naomi Simone Borwein, Richard P. Brent, Regina S. Burachik, Judy-anne Heather Osborn, Brailey Sims, Qiji J. Zhu
PublisherSpringer Verlag
Pages233-253
Number of pages21
Volume313
ISBN (Electronic)9783030365684
ISBN (Print)9783030365677
DOIs
Publication statusPublished - 2020
EventJonathan Borwein Commemorative Conference, JBCC 2017 - Newcastle, Australia
Duration: 25 Sep 201729 Sep 2017

Publication series

NameSpringer Proceedings in Mathematics and Statistics
ISSN (Print)2194-1009
ISSN (Electronic)2194-1017

Conference

ConferenceJonathan Borwein Commemorative Conference, JBCC 2017
CountryAustralia
CityNewcastle
Period25/09/1729/09/17

Keywords

  • Analytical significance
  • Causality
  • Herding
  • Mental models
  • P-values
  • Power relationships
  • Size effects
  • Statistical significance
  • Theory

Fingerprint Dive into the research topics of 'A holistic approach to empirical analysis: The insignificance of P, hypothesis testing and statistical significance'. Together they form a unique fingerprint.

Cite this