Models are often evaluated when their behavior is at its closest to a single, sometimes averaged, set of empirical results, but this evaluation neglects the fact that both model and human behavior can be heterogeneous. Here, we develop a measure, g-distance, which considers model adequacy as the extent to which models exhibit a similar range of behaviors to the humans they model. We define g as the combination of two easily interpretable dimensions of model adequacy: accommodation and excess flexibility. We apply this measure to five models of an irrational learning effect, the inverse base-rate effect (IBRE). g-distance identifies two models, a neural network with rapid attentional shifts (NNRAS) and a dissimilarity-similarity generalized context model (DGCM18), that outperform the previously most supported model (EXIT). We show that this conclusion holds for a wide range of beliefs about the relative importance of excess flexibility and accommodation. We further show that a pre-existing metric, the Bayesian Information Criterion (BIC), misidentifies a known-poor model of the IBRE as the most adequate model. Along the way, we discover that some of the models accommodate human behavior in ways that seem unintuitive from an informal understanding of their operation, thus underlining the importance of formal expression of theories. We discuss the implications of our findings for model evaluation generally, and for models of the inverse base-rate effect in particular, and end by suggesting future avenues of research in computational modeling.
@article{dome2025gdistance,title={g-distance: On the comparison of model and human heterogeneity},publisher={PsyArXiv},journal={Psychological Review.},author={Dome, Lenard and Wills, Andy},year={2025},doi={https://doi.org/10.31234/osf.io/ygmcj},notes={accepted.},}
Lenard Dome, and Andy Wills (2025) Better generalization through distraction? Concurrent load reduces the size of the inverse base-rate effect.Psychonomic Bulletin & Review. accepted.
The inverse base-rate effect (IBRE) is an irrational phenomenon in predictive learning. It occurs when people try to generalize what they have experienced to novel and ambiguous events. This irrational generalization manifests as a preference for rare, unlikely outcomes in the face of ambiguity. A formal mathematical model of this irrational preference leads to a counter-intuitive prediction: the effect disappears under concurrent load. We tested this prediction across two experiments (𝑁1 = 72, 𝑀𝑎𝑔𝑒 = 20.12; 𝑁2 = 160, 𝑀𝑎𝑔𝑒 = 20.88). We confirm the prediction, but only when participants were under an obvious time constraint. This empirical confirmation is as surprising as the prediction itself—irrationality reduces under increased task demands.
@article{dome2025better,title={Better generalization through distraction? Concurrent load reduces the size of the inverse base-rate effect},journal={Psychonomic Bulletin & Review.},author={Dome, Lenard and Wills, Andy},year={2025},doi={10.31234/osf.io/eskr9},notes={accepted.},}
Lenard Dome, and Andy J. Wills (2023) Errorless irrationality: removing error-driven components from the inverse base-rate effect paradigm.In Proceedings of the Annual Meeting of the Cognitive Science Society 45 (45) 237-243
The inverse base-rate effect is a robust irrational bias that arises when people face ambiguity. The most prominent theories of this irrational bias depend on prediction error. In this study, we gradually removed elements of a predictive learning design to test the extent to which error-driven processes underlie this bias. In our first experiment, we removed explicit feedback by implementing the inverse base-rate effect in an observational learning procedure. In our second study, we further removed any causal relationship between stimulus features and category labels by moving towards an unsupervised learning procedure. This removed any information participants could use to identify category labels. In both experiments, the inverse base-rate effect persisted and remained robust. This outcome suggests that this irrational bias is independent of supervised learning procedures. We propose that any theories and models of the inverse base-rate effect must manage information encoding and connection updates without explicit prediction error. We end by proposing two clear paths for future investigations.
@inproceedings{dome2023errorless,title={Errorless irrationality: removing error-driven components from the inverse base-rate effect paradigm},author={Dome, Lenard and Wills, Andy J.},booktitle={Proceedings of the Annual Meeting of the Cognitive Science Society},volume={45},number={45},year={2023},doi={https://doi.org/10.31234/osf.io/936bj},pages={237-243},}
Paper Alert!!! Dome and Wills (2023) Errorless irrationality: removing error-driven components from the inverse base-rate effect paradigm. In Proceedings of the Annual Meeting of the Cognitive Science Society 45 (45) 237-243
2023.04.18.
Version 1.0 of catlearn now available for download from CRAN. It has been downloaded 37,000 times.
2021.06.07.
psp is released! n-dimensional parameter space partitioning algorithm for evaluating formal computational models.