Choosing a Statistical Test: A Decision Guide by Question, Variables and Design

Most problems with statistical tests do not come from arithmetic; software handles that. They come from a mismatch between the test and the question. A student compares three groups with several t-tests instead of one ANOVA, runs a Pearson correlation on a five-point item treated as continuous without justification, or analyzes before-and-after scores as if they came from different people. Each mistake produces a number that looks authoritative but answers the wrong question.

The safest approach is to decide on the analysis before you collect data, ideally in your proposal or pre-registration, and to write down why. That turns the choice into a reasoned decision rather than a search for the test that gives a significant result. The sections below follow the order in which an experienced analyst usually thinks.

Almost every quantitative research question falls into one of a handful of families. Identifying yours narrows the options dramatically.

Describing means summarizing a sample: means, medians, proportions and their spread. No hypothesis test is needed, though confidence intervals help. Comparing means asking whether groups or conditions differ on an outcome, for example whether exam scores differ between teaching methods. Associating asks whether two variables move together, such as study hours and grades. Predicting or explaining asks how much one or more variables account for an outcome while holding others constant, which is the territory of regression. Testing agreement or fit asks whether observed frequencies match expectations or whether two raters agree.

• Difference questions: 'Do nurses on night shifts report higher fatigue than day-shift nurses?'
• Association questions: 'Is social media use related to sleep quality?'
• Prediction questions: 'Do price, trust and ease of use predict purchase intention?'
• Frequency questions: 'Is preferred study mode independent of gender?'
• Time-to-event questions: 'How long until patients are readmitted, and what shortens that time?'

Next, classify the outcome (dependent) variable and the predictor or grouping (independent) variables. The outcome's type usually matters most, because it determines the family of test or model.

Continuous variables are measured on a scale with meaningful distances, such as reaction time, income or blood pressure. Ordinal variables have ordered categories without equal spacing, such as a single Likert item or education level. Nominal variables are unordered categories, such as country or treatment group; a binary variable is a nominal variable with two categories. Counts, such as the number of hospital visits, and time-to-event outcomes have their own dedicated models.

Likert-type data deserve a note. A single item with five ordered options is ordinal. A scale score averaged or summed from several items is often treated as approximately continuous, which is a common and usually defensible practice, provided the scale is reliable and the distribution is not extreme. State which you are doing and why.

The third question is how the observations relate to each other. Independent groups contain different people, as in a comparison between two schools. Related (paired or repeated) measures come from the same people measured more than once, or from matched pairs, as in a pre-test and post-test. Treating related data as independent throws away information and usually violates the independence assumption; treating independent data as paired is simply wrong.

Also count the number of groups or conditions (two, or three or more) and the number of factors. A design with two factors, such as teaching method and gender, calls for a factorial ANOVA or a regression with an interaction rather than two separate tests. Data nested within clusters, such as students within classes or repeated measures within patients, may need multilevel or mixed-effects models, because observations within a cluster are not independent.

Parametric tests such as t-tests, ANOVA, Pearson correlation and linear regression rest on assumptions. The most important are independence of observations, approximate normality of the residuals (not of the raw variable), homogeneity of variance across groups, and, for regression, linearity and absence of severe multicollinearity. Outliers can distort all of these.

Check assumptions with plots first: histograms and Q-Q plots of residuals, boxplots by group, and residual-versus-fitted plots. Formal tests such as Shapiro-Wilk or Levene's test are useful supplements, but they become oversensitive with large samples and underpowered with small ones, so do not let a single p-value decide.

When an assumption fails, you have several options, and the best one depends on which assumption and how badly. Welch's t-test and Welch's ANOVA handle unequal variances well and many statisticians now recommend Welch's t-test as the default for two independent groups. Robust standard errors address heteroscedasticity in regression. Transformations can help with skewed outcomes, though they change the interpretation. Non-parametric tests, such as Mann-Whitney U or Kruskal-Wallis, compare distributions or ranks rather than means, so they answer a slightly different question. Bootstrapping offers another route to confidence intervals without strong distributional assumptions.

Use the table below once you know your question type, outcome type and design. It covers the tests used in the large majority of dissertations. The alternative column lists what to consider if key assumptions are clearly violated or the outcome is ordinal.

Example one: a master's student asks whether a mindfulness program reduces stress. Thirty participants complete a validated stress scale before and after the program. The outcome is a scale score (treated as continuous) and the design is repeated measures with two time points, so the paired-samples t-test fits. If the distribution of difference scores is strongly skewed with a small sample, the Wilcoxon signed-rank test is the fallback. Adding a control group changes the design to a mixed ANOVA, or better, an ANCOVA on post-test scores controlling for pre-test scores.

Example two: a public health researcher asks whether smoking status (smoker or non-smoker) is associated with a diagnosis of hypertension (yes or no) in a survey of adults. Both variables are binary and independent, so a chi-square test of independence answers the association question. If the researcher also wants to control for age and body mass index, the question has become a prediction question with a binary outcome, and binary logistic regression is the appropriate model.

Example three: a business student wants to know whether perceived usefulness, trust and price predict intention to use a mobile banking app, measured on multi-item scales. Multiple linear regression answers this if each construct is represented by a scale score. If the student wants to model the constructs as latent variables with measurement error, structural equation modeling becomes the better tool.

Choosing the right test is only half the job. Modern reporting standards, including the APA Publication Manual (7th edition) that many social science departments follow, expect effect sizes and confidence intervals alongside p-values. An effect size tells readers how large a difference or relationship is; a p-value only tells them how surprising the data would be if there were no effect.

Report the test statistic with its degrees of freedom, the exact p-value (for example p = .032 rather than p < .05, except for very small values), an effect size with a confidence interval, and descriptive statistics for each group. Common effect sizes include Cohen's d or Hedges' g for mean differences, eta-squared or partial eta-squared for ANOVA, r for correlations, R-squared for regression models, and odds ratios for logistic regression.

If you run many tests, consider how you control the chance of false positives. Post hoc procedures such as Tukey's HSD, or corrections such as Holm or Bonferroni, address this for pairwise comparisons. For a set of pre-specified hypotheses, explain your approach in the methods section. Turning numbers into a clear narrative is its own skill, and guidance on results interpretation and reporting can help you present them accurately.

• State the test and why it fits the question and design.
• Describe how assumptions were checked and what you did when they failed.
• Report effect sizes with confidence intervals, not only p-values.
• Keep interpretation proportional: a non-significant result is not proof of no effect.
• Make analysis reproducible by saving syntax (for example SPSS syntax or R scripts).

These errors recur in dissertations across disciplines. Most are easy to fix once you know to look for them.

• Running multiple t-tests instead of ANOVA or a model, which inflates the false-positive rate.
• Testing normality of the raw outcome instead of the residuals or within-group distributions.
• Switching to a non-parametric test automatically, without noting that it answers a different question.
• Ignoring clustering (students in classes, patients in hospitals), which makes standard errors too small.
• Dichotomizing a continuous variable (for example 'high' versus 'low' at the median), which wastes information.
• Interpreting correlation or regression coefficients from cross-sectional data as causal effects.
• Choosing the test after seeing which one gives p < .05.

Some research questions outgrow the classic tests. Longitudinal data with many time points, unbalanced designs, missing data and nested structures are often better handled by linear mixed models or generalized estimating equations. Latent constructs measured by several items may call for factor analysis or structural equation modeling. Panel data in economics and finance have their own family of fixed and random effects models. None of these are exotic, but they require more planning and clearer justification.

If you are unsure, sketch your data structure on paper: what is one row, what is measured on each row, and which rows belong together. That sketch usually reveals whether a simple test is adequate or whether a model is needed. Discuss it with your supervisor or a statistician before collecting data, when changes are still cheap.