Rating traits together or apart: Presentation format affects first impression judgements

In perception experiments, researchers often collect multiple perceptual judgements from the same stimuli and participants to answer their research questions. While the way in which perceptual judgements are collected may affect the data, research rarely considers task design when investigating this topic. In our experiment, we investigated how two frequently used ways of collecting multiple ratings for affect perceptual judgements, focusing on first impressions of faces. One participant group provided ratings of seven person characteristics (e.g. femininity, youthfulness, trustworthiness) blocked by characteristic, rating faces for one person characteristic at a time, across seven blocks. Another participant group completed one block in which they rated all seven characteristics simultaneously via a list of rating scales. The listed presentation format reduced task duration by 22%, but affected the perceptual ratings in several ways, pointing to reduced data quality. Inter-rater agreement was lower for some person characteristics (youthfulness, femininity, and trustworthiness) in the listed format. Variance in ratings was also reduced for youthfulness, femininity, and dominance, with ratings clustering closer to the middle of the scale. Importantly, correlations between different person characteristics became universally positive in the listed format, indicating reduced independence of judgments. These findings highlight that while presenting multiple judgements at the same time may offer efficiency, this approach can introduce systematic biases and potentially reduce the reliability of perceptual data. We therefore suggest using a blocked presentation format and consider how these trade-offs would impact experiments looking at multiple perceptual judgements collected from the same participants.