I just submitted a paper on hidden multiplicity (i.e., a problem of multiple comparisons) in multiway ANOVA. In brief, when a researcher uses a multiway ANOVA without specifying hypotheses a priori, we argue that this researcher is in what Adriaan de Groot (1969) called the “guess” phase of the empirical cycle in which hypotheses are formed rather than tested. And in that case, for a design with, say, two factors, we argue that the family of hypotheses encompasses all hypotheses implied by the design (in this case three: two main effects and one interaction). This inflates the family-wise Type I error to 14% instead of 5% as a result of which the alpha level should be corrected accordingly (multiple methods exist to do this, for example the sequential Bonferroni procedure). By reviewing the 2010 volumes of six flagship journals in psychology, we show that applied researchers are generally not aware of this problem; that they, as a result, do not correct the alpha level; and that applying the sequential Bonferroni correction to a random subsample of articles alters at least one of the substantive conclusions in 45 out of 60 articles considered. We conclude that researchers either have to correct alpha levels when using multiway ANOVA to “see what we can find” (de Groot, 1969), or should preregister their hypotheses. In the latter case, one enters the “predict” phase of the empirical cycle in which hypotheses are tested. In such a confirmative setting, each hypothesis could be argued to constitute its separate family; hence obviating the need to correct the alpha level.
Here’s an interview (in Dutch) on Radio 1 about my dissertation.
Here is an article (in Dutch) about my dissertation, on Kennislink: a site dedicated to disseminating scientific knowledge to a broad audience.
Yesterday (September1, 2013) I was interviewed in the Labyrint radioshow (in Dutch) about my dissertation. See here for the interview online!
On June 13, 2013, I received the IOPS (Interuniversity Graduate School of Psychometrics and Sociometrics) Best Paper Award 2012 for the paper “Dimensions of normal personality as networks in search of equilibrium: You can’t like parties if you don’t like people”; which was published in the European Journal of Personality. See IOPS website: http://www.iops.nl/category/home