contact@mightymetrika.com
Informative Hypothesis Testing
In its essence, informative hypothesis testing (IHT) is a familiar concept to most who are familiar with the t-test. A two-tailed t-test can be used to test:
Ho: sample mean = 10
Ha: sample mean ≠ 10
If the test is conducted at the 0.05 alpha level, then we will reject the null hypothesis if:
- The mean is in the lower 2.5% of the distribution
- The mean is in the upper 2.5% of the distribution
The two-tailed test is simultaneously testing if if the mean is significantly greater the 10 and if the mean is significantly less than 10. On the other hand, a one-tailed t-test can be used to test (we choose mean > 10 for this example but we could also work an example with mean < 10):
Ho: sample mean = 10
Ha: sample mean > 10
If the test is conducted at the 0.05 alpha level, then we will reject the null hypothesis if:
- The mean is in the upper 5% of the distribution
The one-tailed test which focuses on the upper tail of the distribution will miss a significant effect associated with the lower tail of the distribution. The one-tailed test gains statistical power at the expense of potentially missing a significant effect in the untested tail of the distribution.
A one-tailed t-test is an example of an IHT.
When is IHT Appropriate?
What I fear is a research project that does not conduct a proper power analysis before data collection and then, upon realizing that the sample size is less than ideal, a decision is made to use a one-tailed t-test rather than a two tailed t-test. With this fear in mind, I like the stats.oarc.ucla advice:
"If you consider the consequences of missing an effect in the untested direction and conclude that they are negligible and in no way irresponsible or unethical, then you can proceed with a one-tailed test."
Let's also consider the issue from a more optimistic perspective. As a working statistician, I often work with researchers, clinicians, and professionals who have a lot of experience and intuition when it comes to reasoning about the qualitative properties of the variables' distributions, correlations between pairs of variables, and how variables change over time under a condition of interest. With this experience and intuition in mind, I like what Vanbrabant and Rosseel (2020) explain:
"In many psychological fields, researchers have specific expectations about the relation between the means of different groups or between (standardized) regression coefficients...In observational studies, researchers often have clear ideas about whether the direction of the effects is positive or negative (see, for example, Richardson, Abraham, & Bond, 2012), indicated by symbols like “<” and “>”. Testing such specific expectations directly is known under various names, such as one-sided testing, order-constrained hypothesis testing, constrained statistical inference, and informative hypothesis testing."
Personally, if I am to use IHT, I would want to pre-register the study so that I can specify that I am using IHT and specify the "constraints" (see below for more on constraints) data collection begins. This would allow the sample size estimation to be informed by the IHT.
Extending IHT
The concept of IHT can be extended to linear regression, ANOVA, generalized linear models, robust regression, mixed effects models, structural equation models and more. In the R programming language, software is available for implementing IHT in both the frequentist and the Bayesian frameworks of statistics. I personally use the following in each each domain:
- Frequentist: restriktor
- Bayesian: bain
Click on one of the above links to learn more about implementing IHT in R. Both links have examples, tutorials, and literature that will get you up and running with IHT. While there are important differences between the software packages, both of them use a similar workflow:
- Fit a statistical model in the regular fashion
- Specify a constraint (i.e., "Group1 > Group2")
- Pass the fitted model and the constraints to a specialized IHT function
- Interpret the results
