In psychology, there is no such thing as a perfect experiment. Often, it is clear from the get-go that there are certain problems (“confounds”) you will not be able to eliminate, no matter how sophisticated your design might be. The remedy is simple and straightforward: measure what you can measure and try to statistically control for these variables. From some enthusiastic applications, I find that the number of covariates appears to be proportional to the degree you want to show off to other researchers how deeply you care about your data. As a psychologist, I can say that we love and embrace covariates (and subtle, yet significant interaction terms, but this practice had some bad press lately). The belief in the statistical procedure of control via covariates thus seems to be deeply rooted in practice, a bit of everyday numerical magic.
Isn’t it just simple regression analysis you may wonder? Yes, it is, but sometimes the simple things in life can turn out to be much more complex. This is one of the hard lessons we routinely learn in science when we are trying to study a seemingly simple behavior in full-blown detail. The same goes for regression analysis, I guess.
Repeatedly, I have made the awkward experience during the review process that authors react very patronizing when I ask about the details of a regression analysis. Even when a regression analysis is all there is terms of analysis in a paper, it might be treated as a boring add-on that slows down the pace of the speculative narrative of the paper. Hence, it must be described in as few definite phrases as possible. Did the authors include an intercept in the model? Did the authors center age? Did the authors add interaction terms between age and something else when they prominently discuss that it follows different age trajectories? You get the gist. Usually, reported tables help (although they may omit intercepts), but often they conflict with the description in the methods, or at least with my understanding of the description. When I get the revisions back, I typically get to read something useless like:
We thank the reviewer for asking these insightful questions about our advanced ‘linear regression’ analysis and are happy to outline step-by-step which buttons we pressed in SPSS. Furthermore, we would like to point out that one of the reviewers is/knows a biostatistician so there is no need to ask for details. We always analyze the data like that and no one ever asks us why.
Running an ANOVA requires no understanding of intercepts and slopes
In fact, I have encountered so many misunderstanding about covariates that I will need to split it into two posts. One key problem is that the way it is taught in statistics lectures, the t-test and sum of squares ANOVA framework seems like an entirely different thing as regression and the general linear model (GLM) to many researchers I have consulted in statistics to date. Likewise, when I submitted papers and tested for group differences using a regression analysis (e.g., male vs. female participants), I had reviewers objecting to use of this ‘unusual method’.
Such a misconception doesn’t have to do any harm though. However, if your intuition about the data and the accompanying model fails you, it is possible to take the wrong track in following up a ‘result’. Or one might not interpret a ‘result’ in the right way. One example is a group analysis of fMRI contrasts including a behavioral covariate.
Let’s suppose we have emotional pictures and motor responses following a different cue, which are supposedly influenced in response speed by the emotional content of the preceding pictures. Now, we enter the contrast images emotional – neutral and the difference in response time between the conditions as our covariate in the GLM. We get a main effect of ‘image category’ in the amygdala. Awesome. Then, we test for an association with the covariate and we get something small…in the amygdala, yes! Okay, well, maybe, not quite in the amygdala, but reasonably close to it. If you overlay the two effects now with a proper threshold, you realize there is no overlap. In situations like that, I have often heard:
Of course, there is no main effect at this spot anymore because all the variance is accounted for by the covariate. That’s obvious, right? You can see that it is still in the vicinity of the main effect so it is basically in the same area.
Controlling for a variable by entering a covariate ≠ shrinking the main effect to 0
This interpretation sounds reasonable. But it is not correct. SPM’s default is to use grand mean centering for covariates. What is left in the contrast for the main effect of ‘image category’ after accounting for the covariate is an ‘intercept’. It is the computed value of the main effect when the covariate is 0, which corresponds to the average of the response time effect after grand mean centering. In general, this does not lead to great changes in the intercept.
In contrast, the alternative description of the result sketched above demonstrates a different intuition about the covariate. There is a certain share of variance represented by the main effect and if I add a ‘good’ covariate, this share will be reduced because the covariate now accounts for it. The stronger the effect of the covariate, the more will the main effect be shrunk towards zero. My guess is that this wrong intuition arises from the teaching of residual plots when linear regression is introduced first. If you think about a regression model only in terms of R2, the intercept probably seems like a leftover that becomes marginalized with improved fit of the model.
Know your stats
What is the implication of such a slight misunderstanding? Certainly, there can be meaningful modulation (read: significant is not sufficient to establish it) by a covariate when there is not a significant main effect in the exact same voxel of the brain. That is not the point. Yet, it is important for the right interpretation. It may suggest that the contrast expends beyond the core area of the main effect in individuals with a higher difference in response time, whereas the other individuals show a more constrained activation. This could be insightful compared to just assuming the main effect is there as well, but stronger in some individuals.
Moreover, these problems also extend to modeling parametric regressors at the first level. Likewise, it is often assumed that the parametric regressor removes all the important variance from the condition regressor. I have frequently heard this ‘explanation’ when I pointed out that regions of the ‘valuation network’ are not part of the main effect of condition. Some regions might even become significantly deactivated when a monetary offer is presented, but less so when the subjective value is high such as the ventromedial prefrontal cortex (see Roy, Shohamy, & Wager, 2012). If you think of the intercept as nothing but a leftover, chances are high that you are missing the big picture of what is going on in the brain.
In the next post, I will explain why some misconceptions about covariates and how they are handled in some statistical software packages (I am looking at you SPSS!) can lead to false inferences and inflated false-positive rates for main effects.
Roy, M., Shohamy, D., & Wager, T. D. (2012). Ventromedial prefrontal-subcortical systems and the generation of affective meaning. Trends Cogn Sci, 16(3), 147-156. doi:10.1016/j.tics.2012.01.005