Joint Models of Neural and Behavioral Data

Psychologists know well that understanding and predicting human behavior are incredibly difficult. Likewise, neuroscientists are acutely aware of how difficult it is to explain how behavior is implemented by neural mechanisms. One tantalizing possibility is that tackling both of these challenges simultaneously may be more tractable than addressing each separately at its own level of analysis. Although this ambitious and integrative path may seem fantastical, if constraints and relationships exist across measures, then this enterprise has hope. To make this hope a reality requires powerful and transparent methods to link measures across levels of analysis in a way that can support inference and model comparison. The authors provide just such a recipe in this book, walking the reader through the motivation, background, math, and code and how to interpret results step-by-step. The solution is joint modeling, which simultaneously analyzes multiple brain and behavioral measures within a shared formal framework, allowing their relationships to inform model fitting and inference. With joint modeling, one can potentially predict behavior in a task better by incorporating brain measures. For example, trial-by-trial fluctuations in EEG signals could indicate how prepared a person is to respond, which could capture variance in response times. Likewise, multiple brain measures could be incorporated to take advantage of the relative strengths of each. For example, fMRI has good spatial resolution but poor temporal resolution, whereas EEG has the mirror pattern of strengths and weaknesses. Joint modeling can pool multiple measures, whether they are behavioral or neural, to improve estimates. When data are missing for a trial, joint modeling can impute the missing value.

In this book, the practicalities of how to do joint modeling are covered, such as how to evaluate whether joint modeling is adding anything on top of considering measures in isolation. The authors start with simple illustrative examples, which include accompanying code, to build the reader’s intuitions and ability to formulate their own models. To appreciate what is possible with joint modeling, a chapter is devoted to considering published examples. Finally, potential solutions to future challenges, such as scaling the approach to more complex problems that involve relating numerous model parameters and brain regions, are discussed.