This essay originally appeared on Substack.

At a conference 2 years ago, I received advice to read the paper The Importance of Falsification in Computational Cognitive Modeling. I have just recently gotten around to reading it, as I’ve been thinking more about what it means to have a useful cognitive model. This question is evergreen, but it’s taken on particular resonance as AI is fundamentally changing the practice of science and what it even means to create scientific knowledge. In my corner of science, AI models are increasingly used 1) directly as cognitive models (in the sense of directly simulating behavioral traces; this is also sometimes referred to as a user simulator or synthetic participant, depending on the application) and 2) to produce intermediate cognitive model artifacts which are then used to generate predictions. To reason more productively about how AI can help us understand the workings of our own minds, it felt worthwhile to spend some time thinking about what “classical” computational cognitive modeling, at its best, can bring to the table.

Which compelled me to read this paper. First, the paper points out that computational cognitive models are commonly evaluated through procedures like computing the likelihood of the data (or a complexity adjusted likelihood metric, like AIC or BIC) under a given model, for several candidate models. The model that achieves the highest likelihood is considered the “best” model. Procedures like these are described as a form of relative evaluation: they result in a stack ranking of different models.

The paper then states that it is less common to actually use the model to simulate behavior, and then evaluate if those simulated outputs capture the patterns in the real human data. This simulation step is important, according to the paper, because it licenses stronger claims; rather than just saying a model is less good, we can actually claim that it doesn’t capture the data and therefore can be falsified. In this sense, simulation is a form of absolute evaluation: it allows you to rule out certain models entirely, if a model fails to capture the behavioral effect of interest. The paper also provides an existence proof that a model can achieve higher likelihood than an alternative, while actually being worse at simulating the behavioral data. The overall recommendation is to pursue both relative evaluation via model-fitting and absolute evaluation via simulation, to realize the core benefit of computational modeling: quantitative predictions which can be falsified.

This is a really interesting point - I wish I had read this paper 2 years ago. This distinction feels particularly relevant as I am exploring ways to make language models into better cognitive models: I am seeing firsthand how achieving high likelihood under the data (e.g. low cross-entropy loss) does not necessarily track with good generative performance (e.g. producing outputs that semantically match the human data).

The broader thesis, however, raises the question: what does it actually mean to falsify a model? Or alternatively, how do you show sort of the opposite - that you have produced “an effect of interest”?

In the (synthetic) example provided in the paper, one model is clearly insufficient at capturing the key qualitative trend; its predictions do not follow the characteristic dips of the ground truth choice rate curve. We could find any number of distance metrics to provide formal backing for an intuitive observation.

But what if the data we are examining is for a fundamentally more complex task, where behavior is expressed not simply as a choice in a 2 way decision, but perhaps as a multi-step trajectory, a natural language utterance, or some other high-dimensional trace? Open-ended tasks surface more naturalistic, everyday behaviors, making them an important testbed for developing more generalizable theories of human behavior. Yet in these settings, it is inherently harder to find a model that captures the data-generating process. It might be harder still if the unit of prediction is individual behavior, rather than some population average, and the modeling problem is even more underconstrained due to limited data. So what if the best model we could identify for this task was one that was just “pretty good”? To pose this more concretely: if this model was mostly accurate in held-out trials, but failed to predict some subset, is it falsified?

The usual defense would be that models predict effects, rather than trial-by-trial behavior. But at some point, if trials are the means through which the theory is tested, then an effect has to be grounded out in trial behavior, leaving the question of falsification unresolved — seemingly more so in settings where behavioral data is high-dimensional.

To avoid these complications, one move might be to codify a procedure for demonstrating falsification. This is in part the logic of posterior predictive checks, in which the fitted model is used to generate synthetic data and the simulated data is then compared to the ground truth as a way to appraise model quality. One can then use statistical testing to determine if there is a significant difference between, say, the mean of the true distribution and that of the simulated distribution; models that produce outputs that are significantly different from the ground-truth might be deemed falsified. But it is far less clear how to do this for classes of behavior that are not well-modeled as a distribution over a single scalar random variable, things other than reaction times or decisions from a fixed set.

In short, in more complex and naturalistic domains, something resembling “absolute” falsification is commensurately harder. The concern is that prioritizing this notion of falsifiability structurally incentivizes highly stylized tasks and simple dependent variables, where it is easier to intuitively and formally appraise the extent to which a model’s simulated outputs match (or fail to match) patterns in the human data. Stated otherwise, falsification as a target can be reward-hacked: when it is presented as an essential box to check, it can come at the cost of other desiderata, like studying behavior in more naturalistic settings.

In any case, I still think the distinction raised in the paper, between a) finding a parameterization of the model that minimizes some loss function and b) simulating behavior under the model and comparing it to actual ground truth data, is a useful and important one. I just think that we need to be clear about the potential trade-offs if “absolute” falsification is prioritized as a goal of cognitive modeling. In particular, there is a risk of de-incentivizing more complex tasks and behaviors where it is less obvious how to demonstrate producing an effect of interest and/or falsification, and consequently drifting further away from the sorts of open-ended behaviors that presumably motivated an experiment in the first place. On the other hand, not explicitly pursuing falsification via simulation limits the scope of claims in a different way, since it is harder to rule out a candidate model, or you might have undue confidence in your best-fitting model (e.g. if it was not put through posterior predictive checking).

This is a trade-off that cannot be bypassed by having AI produce cognitive models (at least at this point in time). It could be the case that AI models have systematic preferences for certain kinds of scientific practices, which would be interesting in its own right. In fact, I suspect that AI, with some scaffolding, will be (if it’s not already) very well-suited to searching through large spaces of cognitive models in relatively constrained tasks and adjudicating between these models, when it is relatively clear how to operationalize falsification. I also remain optimistic that AI can be intentionally steered by creative human scientists to produce new kinds of cognitive models for qualitatively more complex domains (I’ve previously written about how LLMs could be deployed to this end), and new statistical techniques for appraising model fit for more complex forms of data. Maybe this work will be informed by hill-climbing in more toy domains, following the trajectory of progress in AI research itself; maybe we can just proceed there directly.

But at least for now, we can’t circumvent entirely the epistemic trade-offs that arise from pursuing model falsifiability as a goal, no matter who or what is on the interpreting end of cognitive models. AI could help us expand the Pareto frontier, by facilitating discovery of better models and more systematic measurement, in turn making models of behavior in naturalistic tasks both possible and amenable to falsification. But to pursue this, we first have to acknowledge that there exists a frontier at all.

RSS feed