How Do We Do Hypothesis Testing? Jeff Gill,jgill@ucdavis.edu 1 The Current Paradigm:Null Hypothesis Significance Testing The current,nearly omnipresent,approach to hypothesis testing in all of the social sciences is a synthesis of the Fisher test of significance and the Neyman-Pearson hypothesis test.In this "modern"procedure,two hypotheses are posited:a null or restricted hypothesis(Ho)which competes with an alternative or research hypothesis (H)describing two complementary notions about some phenomenon.The research hypothesis is the probability model which describes the author's belief about some underlying aspect of the data,and operationalizes this belief through a parameter:0.In the simplest case,described in every introductory text,a null hypothesis asserts that =0 and a complementary research hypothesis asserts that 00.More generally,the test evaluates a parameter vector:0=101,02,...,0m,and the null hypothesis places restrictions on some subset (e<m)of the theta vector such as: 0i=k10j+k2 with constants ki and k2. A test statistic (T),some function of 0 and the data,is calculated and compared with its known distribution under the assumption that Ho is true.Commonly used test statistics are sample means (X),chi-square statistics (x2),and t-statistics in linear (OLS)regression analysis.The test procedure assigns one of two decisions (Do,D1)to all possible values in the sample space of T,which correspond to supporting either Ho or Hi respectively.The p-value ("associated probability")is equal to the area in the tail (or tails)of the assumed distribution under Ho which starts at the point designated by the placement of T on the horizontal axis and continues to infinity.If a predetermined a level has been specified,then Ho is rejected for p-values less than a,otherwise the p-value itself is reported.Formally, the sample space of T is segmented into two complementary regions (So,S1)whereby the probability that T falls in S1,causing decision D1,is either a predetermined null hypothesis cumulative distribution function (CDF)level:the probability of getting this or some lower value given a specified parametric form such as normal,F,t,etc.(a size of the test, Neyman and Pearson),or the cumulative distribution function level corresponding to the value of the test statistic under Ho is reported (p-value =IPHo(T =t)dt,Fisher).Thus 51 decision D is made if the test statistic is sufficiently atypical given the distribution under Ho.This process is illustrated for a one tail test at a =0.05 in Figure 1. 2 Historical Development The current null hypothesis significance test is synthesis of two highly influential but incom- patible schools of thought in modern statistics.Fisher developed a procedure that produces significance levels from the data whereas Neyman and Pearson posit an intentionally rigid decision process which seeks to confirm or reject specified a priori hypotheses.The null hypothesis significance testing procedure is not influenced by the third major intellectual 1How Do We Do Hypothesis Testing? Jeff Gill, jgill@ucdavis.edu 1 The Current Paradigm: Null Hypothesis Significance Testing The current, nearly omnipresent, approach to hypothesis testing in all of the social sciences is a synthesis of the Fisher test of significance and the Neyman-Pearson hypothesis test. In this “modern” procedure, two hypotheses are posited: a null or restricted hypothesis (H0) which competes with an alternative or research hypothesis (H1) describing two complementary notions about some phenomenon. The research hypothesis is the probability model which describes the author’s belief about some underlying aspect of the data, and operationalizes this belief through a parameter: θ. In the simplest case, described in every introductory text, a null hypothesis asserts that θ = 0 and a complementary research hypothesis asserts that θ 6= 0. More generally, the test evaluates a parameter vector: θ = {θ1, θ2, . . . , θm}, and the null hypothesis places restrictions on some subset (` ≤ m) of the theta vector such as: θi = k1θj + k2 with constants k1 and k2. A test statistic (T), some function of θ and the data, is calculated and compared with its known distribution under the assumption that H0 is true. Commonly used test statistics are sample means (X¯), chi-square statistics (χ 2 ), and t-statistics in linear (OLS) regression analysis. The test procedure assigns one of two decisions (D0, D1) to all possible values in the sample space of T, which correspond to supporting either H0 or H1 respectively. The p-value (“associated probability”) is equal to the area in the tail (or tails) of the assumed distribution under H0 which starts at the point designated by the placement of T on the horizontal axis and continues to infinity. If a predetermined α level has been specified, then H0 is rejected for p-values less than α, otherwise the p-value itself is reported. Formally, the sample space of T is segmented into two complementary regions (S0, S1) whereby the probability that T falls in S1, causing decision D1, is either a predetermined null hypothesis cumulative distribution function (CDF) level: the probability of getting this or some lower value given a specified parametric form such as normal, F, t, etc. (α = size of the test, Neyman and Pearson), or the cumulative distribution function level corresponding to the value of the test statistic under H0 is reported (p-value = R S1 PH0 (T = t)dt, Fisher). Thus decision D1 is made if the test statistic is sufficiently atypical given the distribution under H0. This process is illustrated for a one tail test at α = 0.05 in Figure 1. 2 Historical Development The current null hypothesis significance test is synthesis of two highly influential but incom￾patible schools of thought in modern statistics. Fisher developed a procedure that produces significance levels from the data whereas Neyman and Pearson posit an intentionally rigid decision process which seeks to confirm or reject specified a priori hypotheses. The null hypothesis significance testing procedure is not influenced by the third major intellectual 1