Home

I read a recent correspondence in Nature Human Behavior, provocatively titled “Why social media research has failed policy-makers”, by Tobias Dahl et al.

This essay caught my eye, in part because it is relatively uncommon to see researchers publicly criticize their own fields and research programs (although this is a recent counterexample). But beyond that, its title invited connections to some of the questions I’ve been thinking about, namely how interactions with AI are changing aspects of human cognition and behavior: there are clear metascientific parallels between this emerging line of work and past / ongoing research that examines how social media use affects wellbeing. To be sure, there are many qualitatively new and different phenomena at play with AI that make it far more than “social media 2.0”. But the methodological challenges we face in studying these questions about AI interaction mirror many of the challenges endured by social media researchers - limited access to proprietary data, ill-defined dependent variables, and difficult-to-resolve tradeoffs about the value of highly-controlled but artificial lab studies vs. more naturalistic yet potentially uninterpretable observational analyses. Is there anything we can learn from where social media research has erred?

Before we get there, let’s unpack the diagnosis provided by Dahl et al. First, they argue that most social media wellbeing research actually doesn’t find a negative relationship between time spent on social media and various metrics of wellbeing. But this does not license the conclusion that social media is benign. Rather, this effect could be driven by the cost of missing out (COMO) phenomenon, where adolescents who are restricted from using social media actually fare worse on wellbeing metrics because they cannot partake in a shared social experience. To make causal claims about the effects of social media, there needs to be a proper control condition in which there was an entire group or community of adolescents not on social media but otherwise matched to the “experimental” group of youth on social media - something that is of course very difficult to come by. Without this setup, we are left with a deeply limited and ultimately non-actionable base of evidence.

I’m not fully convinced by the specific COMO explanation offered here - if one is pre-committed to a particular point of view (e.g. social media is or is not harmful), one could concoct any number of confounds that explain why the result apparently goes in the other direction (see this for more examples of how this is playing out in the social media debate). I’m generally sympathetic to claims that social media is a net negative to wellbeing, but it’s unclear to me if COMO is actually a reasonable explanation as to why the evidence looks mixed. But more broadly, I think it’s worthwhile to highlight the importance (and difficulty) of having a reasonable control / baseline group to draw causal claims - a recent essay emphasized this precise point in trying to assess the effects of chatbots in domains such as mental health.

Of course, principles of experimental design are not unique to social media research; this is just one manifestation. So despite my initial optimism that we could learn something from our predecessors, I am left with the impression that little from the social media playbook is actually transferable to AI interaction research, beyond serving as a generic cautionary tale about the difficulty of trying to pursue research when the most illuminating data is proprietary.

To highlight a specific methodological area of disanalogy: this essay seems to suggest that within the social media wellbeing research community, it is a common practice to bundle all interactions with social media into a catch-all “use” or “time spent” quantity. I’m not sure this was ever a great methodological choice, but it seems even less useful to do that when studying the effects of AI, because the set of things one can do with AI is much broader, even compared to social media. Imagine 2 study subjects: one who uses AI to speed through school assignments, versus another who uses AI to create practice problems to then solve independently. Under a coarse-grained “use” metric, these two subjects could very well be classified into the same bucket, which feels obviously wrong. There are inertial forces (e.g. the fact that it’s relatively easy to get self-reports of time spent on a specific AI product) that will likely push researchers toward the catch-all “use” value as a predictor variable, but ultimately that practice does not seem like it should be rolled over.

To me, this case study showcases the limits of analogical thinking about how past technologies have shaped human cognition and behavior. Ultimately, I actually found it a more useful exercise to think about the ways in which AI is not like social media, than to try to draw explicit parallels.

If we hope to arrive at a holistic picture of how AI is changing how we think, there are a lot of hard problems that need to be addressed; social media research didn’t pave the way methodologically, although maybe it offers a useful point of contrast. An additional consolation is that the object of study is so flexible as to be a useful analytical tool to help us answer these questions, more quickly: AI, used judiciously, could potentially help unblock some of these methodological challenges (e.g. coding agents could make it easy to build a more granular tracker of individuals’ interactions with AI). The big question remains, though, of how we will sift through an ever-expanding body of AI-powered research to understand what this technology is doing to our own minds - and if we can draw conclusions at the speed at which AI is infusing itself into our lives.