Uri Simonsohn, Analyzing Experiments With Multiple Stimuli (In The Real World)

Abstract

 

Experimenters in psychology choose stimuli to proxy for underlying theoretical constructs. This selection is typically haphazard, easily resulting in confounded manipulations that threaten studies' validity. We introduce "match-and-mix", a 7-step procedure for stimuli generation, that focuses on confound-management in general and relying on matched-pairs of stimuli across conditions in particular. Then, we note that the current consensus for analyzing experiments with multiple stimuli, complex mixed-models, leads to (1) low power, and (2) missing informative contrasts across specific stimuli. We propose, instead, running simpler analysis (e.g., t tests) on the aggregate data, followed by exploratory visual displays we call "stimuli plots", contrasting observed vs expected variation across fully described stimuli. We make these points re-analyzing data from published papers and conducting simulations.

Elath Hall, 2nd floor, Feldman Building, Edmond Safra Campus