BOOTSTRAP CONFIDENCE INTERVALS 191 TABLE 2 Exact and approximate confidence intervals for the correlation coefficient,ed4 data;=0.723:the bootstrap methods ABC,BCa bootstrap-t and calibrated ABC are explained in Sections 2-7;the ABC and BCa intervals are close to exact in the normal theory situation (left panel);the standard interval errs badly at both endpoints,as can be seen from the coverage probabilities in the bottom rows Normal theory Nonparametric Exact ABC BCa Bootstrap-t Standard ABC BCa Bootstrap-t Calibrated Standard 0.05 0.47 0.47 0.47 0.45 0.55 0.56 0.55 0.51 0.56 0.59 0.95 0.86 0.86 0.86 0.87 0.90 0.83 0.85 0.86 0.83 0.85 Length 0.39 0.39 0.39 0.42 0.35 0.27 0.30 0.35 0.27 0.26 Shape 0.52 0.52 0.54 0.52 1.00 0.67 0.70 0.63 0.67 1.00 Cov 05 0.05 0.05 0.05 0.04 0.12 Cov 95 0.95 0.95 0.95 0.97 0.99 are a random sample ("i.i.d.")from some unknown One of the achievements of the theory discussed bivariate distribution F. in Section 8 is to provide a reasonable theoretical gold standard for approximate confidence inter- (1.7) iid.F, i=1,2,.,n, vals.Comparison with this gold standard shows that the bootstrap intervals are not only asymptot- n =20,without assuming that F belongs to any ically more accurate than the standard intervals, particular parametric family.Bootstrap-based confi- they are also more correct."Accuracy"refers to the dence intervals such as abC are available for non- coverage errors:a one-sided bootstrap interval of parametric situations,as discussed in Section 6.In intended coverage a actually covers 6 with proba- theory they enjoy the same second-order accuracy as bility a+0(1/n),where n is the sample size.This in parametric problems.However,in some nonpara- is second-order accuracy,compared to the slower metric confidence interval problems that have been first-order accuracy of the standard intervals,with examined carefully,the small-sample advantages of coverage probabilites a+0(1//n).However con- the bootstrap methods have been less striking than fidence intervals are supposed to be inferentially in parametric situations.Methods that give third- correct as well as accurate.Correctness is a harder order accuracy,like the bootstrap calibration of an property to pin down,but it is easy to give exam- ABC interval,seem to be more worthwhile in the ples of incorrectness:if x1,x2,...,xn is a random nonparametric framework(see Section 6). sample from a normal distribution N(0,1),then In most problems and for most parameters there (min(x:),max(xi))is an exactly accurate two-sided will not exist exact confidence intervals.This great confidence interval for 6 of coverage probability gray area has been the province of the standard in- 1-1/2"-1,but it is incorrect.The theory of Section tervals for at least 70 years.Bootstrap confidence in- 8 shows that all of our better confidence intervals tervals provide a better approximation to exactness are second-order correct as well as second-order in most situations.Table 3 refers to the parameter accurate.We can see this improvement over the 6 defined as the maximum eigenvalue of the covari- standard intervals on the left side of Table 2.The ance matrix of(B,A)in the cd4 experiment, theory says that this improvement exists also in (1.8) 6=maximum eigenvalue {cov(B,A)}. those cases like Table 3 where we cannot see it directly. The maximum likelihood estimate(MLE)of 0,as- suming either model (1.2)or (1.7),is 6=1.68.The bootstrap intervals extend further to the right than 2.THE BCa INTERVALS to the left of 6 in this case,more than 2.5 times as The next six sections give a heuristic overview far under the normal model.Even though we have of bootstrap confidence intervals.More examples no exact endpoint to serve as a"gold standard"here, are presented,showing how bootstrap intervals the theory that follows strongly suggests the supe- can be routinely constructed even in very compli- riority of the bootstrap intervals.Bootstrapping in- cated and messy situations.Section 8 derives the volves much more computation than the standard second-order properties of the bootstrap intervals in intervals,on the order of 1,000 times more,but the terms of asymptotic expansions.Comparisons with algorithms are completely automatic,requiring no likelihood-based methods are made in Section 9 more thought for the maximum eigenvalue than the The bootstrap can be thought of as a convenient correlation coefficient,or for any other parameter. way of executing the likelihood calculations in para-BOOTSTRAP CONFIDENCE INTERVALS 191 Table 2 Exact and approximate confidence intervals for the correlation coefficient, cd4 data; θˆ = 0:723: the bootstrap methods ABC, BCa , bootstrap-t and calibrated ABC are explained in Sections 2–7; the ABC and BCa intervals are close to exact in the normal theory situation (left panel); the standard interval errs badly at both endpoints, as can be seen from the coverage probabilities in the bottom rows Normal theory Nonparametric Exact ABC BCa Bootstrap-t Standard ABC BCa Bootstrap-t Calibrated Standard 0.05 0.47 0.47 0.47 0.45 0.55 0.56 0.55 0.51 0.56 0.59 0.95 0.86 0.86 0.86 0.87 0.90 0.83 0.85 0.86 0.83 0.85 Length 0.39 0.39 0.39 0.42 0.35 0.27 0.30 0.35 0.27 0.26 Shape 0.52 0.52 0.54 0.52 1.00 0.67 0.70 0.63 0.67 1.00 Cov 05 0.05 0.05 0.05 0.04 0.12 Cov 95 0.95 0.95 0.95 0.97 0.99 are a random sample (“i.i.d.”) from some unknown bivariate distribution F, 1:7‘  Bi Ai  ∼i:i:d: F; i = 1; 2;: : :; n; n = 20, without assuming that F belongs to any particular parametric family. Bootstrap-based confi- dence intervals such as ABC are available for non￾parametric situations, as discussed in Section 6. In theory they enjoy the same second-order accuracy as in parametric problems. However, in some nonpara￾metric confidence interval problems that have been examined carefully, the small-sample advantages of the bootstrap methods have been less striking than in parametric situations. Methods that give third￾order accuracy, like the bootstrap calibration of an ABC interval, seem to be more worthwhile in the nonparametric framework (see Section 6). In most problems and for most parameters there will not exist exact confidence intervals. This great gray area has been the province of the standard in￾tervals for at least 70 years. Bootstrap confidence in￾tervals provide a better approximation to exactness in most situations. Table 3 refers to the parameter θ defined as the maximum eigenvalue of the covari￾ance matrix of B; A‘ in the cd4 experiment, 1:8‘ θ = maximum eigenvalue covB; A‘•: The maximum likelihood estimate (MLE) of θ, as￾suming either model (1.2) or (1.7), is θˆ = 1:68. The bootstrap intervals extend further to the right than to the left of θˆ in this case, more than 2.5 times as far under the normal model. Even though we have no exact endpoint to serve as a “gold standard” here, the theory that follows strongly suggests the supe￾riority of the bootstrap intervals. Bootstrapping in￾volves much more computation than the standard intervals, on the order of 1,000 times more, but the algorithms are completely automatic, requiring no more thought for the maximum eigenvalue than the correlation coefficient, or for any other parameter. One of the achievements of the theory discussed in Section 8 is to provide a reasonable theoretical gold standard for approximate confidence inter￾vals. Comparison with this gold standard shows that the bootstrap intervals are not only asymptot￾ically more accurate than the standard intervals, they are also more correct. “Accuracy” refers to the coverage errors: a one-sided bootstrap interval of intended coverage α actually covers θ with proba￾bility α + O1/n‘, where n is the sample size. This is second-order accuracy, compared to the slower first-order accuracy of the standard intervals, with coverage probabilites α + O1/ √ n‘. However con- fidence intervals are supposed to be inferentially correct as well as accurate. Correctness is a harder property to pin down, but it is easy to give exam￾ples of incorrectness: if x1 ; x2 ;: : :; xn is a random sample from a normal distribution Nθ; 1‘, then (minxi ‘, maxxi ‘) is an exactly accurate two-sided confidence interval for θ of coverage probability 1 − 1/2 n−1 , but it is incorrect. The theory of Section 8 shows that all of our better confidence intervals are second-order correct as well as second-order accurate. We can see this improvement over the standard intervals on the left side of Table 2. The theory says that this improvement exists also in those cases like Table 3 where we cannot see it directly. 2. THE BCa INTERVALS The next six sections give a heuristic overview of bootstrap confidence intervals. More examples are presented, showing how bootstrap intervals can be routinely constructed even in very compli￾cated and messy situations. Section 8 derives the second-order properties of the bootstrap intervals in terms of asymptotic expansions. Comparisons with likelihood-based methods are made in Section 9. The bootstrap can be thought of as a convenient way of executing the likelihood calculations in para-