A confidence interval estimates a range of plausible values for a parameter, while a hypothesis test evaluates whether observed data are compatible with a specific null value. The two are complementary, not competing: a two-sided test at significance level α rejects the null exactly when the (1 − α) confidence interval excludes it. This duality is the cleanest way to see why reporting only a p-value throws away information a confidence interval preserves.
The shared machinery underneath both tools
Both procedures start from the same place: a sample statistic, its assumed sampling distribution, and a standard error. A hypothesis test asks where the observed statistic falls relative to a null-hypothesized distribution. A confidence interval inverts that calculation, asking which null values would not be rejected at the chosen level.
The inversion is exact for symmetric, pivotal statistics. For a one-sample mean with known variance, the 95% confidence interval for μ is the sample mean plus or minus 1.96 standard errors. The corresponding two-sided test rejects a null mean of μ₀ at α = 0.05 precisely when μ₀ sits outside that interval. Same arithmetic, different output.
Two quantities drive the width of every interval and the sensitivity of every test: the standard error (shrinking with the square root of sample size n) and the confidence level or alpha. Doubling n halves the standard error, roughly halving interval width. Raising confidence from 95% to 99% widens the interval because the critical value rises from about 1.96 to 2.58 for a normal approximation.
A hypothesis test returns an answer to one narrow question. A confidence interval returns a magnitude, a direction, and a precision estimate in a single object. That difference is the reason modern reporting guidance favors intervals over bare p-values.
Quantified context: what the numbers actually look like
Consider a two-arm trial with a mean difference of 4.2 units and a 95% confidence interval of 0.5 to 7.9 units. The corresponding two-sided test yields p ≈ 0.027. The interval excludes zero, the test rejects the null, and the two agree, as they must at α = 0.05.
Now shift the interval to −0.8 to 8.2 units with the same point estimate. The test now produces p ≈ 0.11 and fails to reject zero. The interval straddles the null. The point estimate did not move; only the standard error changed. This illustrates why sample size, not the observed effect, often decides statistical significance.
Wider still, an interval of −3.0 to 11.4 units with p ≈ 0.25 is uninformative about direction but still bounds the plausible effect. A reader learns that effects larger than roughly 11 units are unlikely, even though the data cannot exclude zero.
Effect-size thresholds matter here. A study can show a statistically significant difference of 0.3 units with a tight interval in a very large sample, yet that effect may fall well below any clinically meaningful threshold. Statistical significance and practical importance are separate judgments, and the interval is the tool that exposes the gap.
The unit system here is US throughout. Where money enters published work as charges or costs, treat figures as study-specific and not transferable.
Reading the interval as the primary result
A confidence interval answers a question a test cannot: how large is the effect likely to be? An interval of 0.5 to 7.9 units says the true effect plausibly lies anywhere in that band. The lower edge near zero signals genuine uncertainty about whether the benefit is trivial, even though the test called the result significant.
The interval also encodes the decision rule at every null value simultaneously. With a 95% interval of 0.5 to 7.9, an investigator can check any hypothesized value without recalculating a p-value. A null of 3.0 would not be rejected at α = 0.05 because 3.0 sits inside the interval. A null of 9.0 would be rejected because 9.0 sits outside.
Precision is visible in the width. Narrow intervals indicate a well-estimated effect; wide intervals warn that the sample cannot resolve the question. Reporting the width alongside the point estimate is what separates a usable result from a number without context.
One professional caveat: the interval's coverage claim is about the procedure, not any single realized interval. A 95% method captures the true parameter in 95% of repeated samples. Any one computed interval either contains the truth or does not, and the analyst never knows which.
Comparison table: choosing between the two framings
| Question asked | Hypothesis test output | Confidence interval output | Best used when | Weakness |
|---|---|---|---|---|
| Is the effect different from null? | p-value and reject/fail decision | Whether null value lies inside the range | A binary regulatory or screening decision | p-value hides effect size |
| How large is the effect? | Not answered | Point estimate plus plausible range | Communicating magnitude to decision-makers | Requires choosing a confidence level |
| How precise is the estimate? | Implied by p-value but not shown | Interval width directly | Comparing studies of differing sample size | Wide intervals can be uninformative |
| Which null values are compatible? | One null at a time | All nulls at once via inversion | Sensitivity analysis across thresholds | Only exact for standard pivotal statistics |
| What about equivalence or non-inferiority? | Requires a margin-based test | Must be paired with a margin | Regulatory submissions | Plain interval alone cannot decide equivalence |
The table's practical message: the two tools answer different questions, and mature reporting shows both. A p-value without an interval hides the effect size. An interval without a stated confidence level hides the error rate the reader should assume.
Edge cases and the assumptions that break the duality
The exact inversion between test and interval holds cleanly for symmetric, pivotal statistics such as the normal-mean case. It frays for asymmetric statistics. Confidence intervals for odds ratios, risk ratios, and proportions near 0 or 1 are typically built on log or transformed scales, so a test p-value and an interval may appear to disagree in borderline cases if computed by different methods.
Small samples compound the problem. With n below roughly 30 and unknown variance, the t-distribution widens the critical value, from about 2.26 at n = 10 to 1.96 as n grows large. Using a normal critical value in a small sample understates the interval and inflates false rejections.
Multiple comparisons break the mapping entirely. If an analyst tests 20 independent null hypotheses at α = 0.05, the probability of at least one false rejection approaches 1 − 0.95²⁰ ≈ 0.64. An unadjusted interval for a single comparison says nothing about that family. Adjusted intervals or Bonferroni-style corrections must accompany family-wise claims.
Dependent data, clustered sampling, and nonresponse all distort the standard error that both tools rely on. When the standard error is wrong, both the test and the interval are wrong in the same direction. Fixing the interval does not fix the assumption; the assumption must be checked first.
Bullet warnings for analysts:
Do not compare overlapping confidence intervals between two groups as a substitute for a formal between-group test; overlap and non-significance are not equivalent. Do not read a 95% interval as a 95% probability that the true value lies inside this specific range. Do not treat a p-value just below 0.05 as evidence of a meaningful effect; check the interval's lower bound against your practical threshold. Do not mix methods; if the test uses a t-distribution, the interval must too. Do not report an interval without its confidence level and sample size.
Common misconceptions that survive peer review
The most persistent error treats a p-value as the probability the null is true. It is not. The p-value is the probability of data at least as extreme as observed, assuming the null holds. A p of 0.03 does not mean a 3% chance the null is correct, and a p of 0.30 does not mean the null is 30% likely.
The second error reads "fail to reject" as "the null is true." Absence of evidence is not evidence of absence. A wide interval spanning zero is compatible with a true null, but it is equally compatible with effects on either side. Only a sufficiently narrow interval centered near zero supports an equivalence claim, and that requires a prespecified margin.
A counterintuitive detail worth flagging: an interval can be entirely on one side of zero while the corresponding test at α = 0.05 fails to reject, if the interval was computed at a different confidence level than 95%. Reporting a 90% interval alongside a 5% test, or vice versa, invites apparent contradictions that are pure arithmetic.
A related trap: p-values and intervals both depend on assumptions that are rarely tested. Normality, independence, and correct model specification can all be violated while the output looks clean. The interval's apparent precision can create unwarranted trust in a misspecified model.
Practical actions: what to compute and report
Start with the question. If a binary regulatory or screening decision is the goal, run the test and report the p-value with the exact null and alternative. If magnitude and precision matter, compute and report the interval at the same confidence level.
Match the confidence level to the test alpha. A two-sided test at α = 0.05 pairs with a 95% interval. A one-sided test at α = 0.05 pairs with a 90% two-sided interval, and reporting that pairing explicitly prevents the apparent contradiction described above.
Reconcile thresholds before interpreting. If a practical significance threshold exists, for example a minimum effect of 2.0 units, check whether the interval's lower bound clears it. An interval of 0.5 to 7.9 straddles the minimum and therefore cannot rule out an effect too small to matter, regardless of the p-value.
Report the interval with its level, the point estimate, the sample size, and the method used to construct it. State whether the interval is symmetric or transformed, and name any multiplicity adjustment applied to a family of comparisons.
For small samples, use t-based or exact methods rather than normal approximations, and note the assumption. For asymmetric outcomes, report intervals on the transformed scale or use methods designed for the parameter of interest.
Prespecify the primary analysis. Post hoc interval computation and post hoc null selection inflate error rates and invalidate the stated confidence level. The interval's coverage guarantee assumes the analysis plan was fixed before data collection.
Limitations of the framing and when neither tool suffices
Both tools assume a sampling framework that is often only approximately true. Nonprobability samples, convenience samples, and administrative data do not support the coverage interpretation of an interval, even when the arithmetic is correct. The output looks identical; the meaning is not.
Neither tool addresses causality. A tight interval around an association says nothing about whether an intervention would move the outcome. Causal claims require design features such as randomization or explicit identification assumptions, and no confidence level repairs a confounded comparison.
Neither tool resolves practical significance, and neither replaces judgment. Statistical significance with a tight interval around a trivial effect is common in large datasets, while a wide interval around a large effect in a small study is equally common. The interval exposes the tension; the analyst must weigh it against domain thresholds.
Finally, the confidence interval and hypothesis test are tools of estimation and decision, not proof. Reporting both, with the assumptions and thresholds stated plainly, is the standard the evidence supports. Anything less asks the reader to guess what the numbers mean.
Whether you report a confidence interval or a hypothesis test depends on the question you are answering. Report both when both matter, and state the confidence level, the null, and the threshold so a reader can reconcile them without guessing.