A holistic approach to empirical analysis: The insignificance of P, hypothesis testing and statistical significance

It is well documented that academics and practitioners focus on statistical significance (typically represented by P tests) and statistical hypothesis testing to determine if their non-statistical analytical hypothesis is correct or likely to be correct. Moreover, statistical significance is relied upon to determine which variables should be used in their models or analyses. In spite of ongoing criticism, this practice continues to the detriment of robust scientific analysis. I discuss the significant limitations of statistical significance in scientific analysis, irrespective of discipline, with some focus on economics. I place statistical significance in a broader analytical context, discussing other analytical procedures that need to be followed and emphasized for one’s analysis to be scientifically robust. This relates the development of models, assumptions underlying the models, the data collected and constructed, the relationship between statistical significance and causality and the importance of non-statistical theory to the identification of pertinent modelling variables. Analytical significance (size effects and variability) is core to any robust scientific analysis, but only in the context of all of the other prior steps in the applied research project being in place. In this broader analytical framework, the statistical significance becomes relatively insignificant. I also address why statistical significance and statistical hypothesis testing dominates the applied analytical landscape even though this dominance is not best practice. Of critical importance are the mental models of best practice and the worldview and preferences of decision makers determining what gets published and who is successful in securing grants and employment.