Machine Learning and Data-Driven Exploration for Longitudinal Data

31.1 Two Cultures, One Researcher

Breiman (2001) drew a line between two cultures of statistical practice. The data-modeling culture posits a stochastic model relating inputs to outputs, estimates its parameters, and interprets them; its currency is explanation, and its validity rests on the model being approximately correct. The algorithmic-modeling culture treats the data-generating mechanism as unknown and unknowable, and seeks any function, however uninterpretable, that predicts the output accurately on new data; its currency is predictive skill, and its validity rests on honest out-of-sample evaluation. Psychology has lived almost entirely in the first culture, and Yarkoni and Westfall (2017) argued that this has cost the field, that an overinvestment in explanation and a corresponding neglect of prediction has produced theories that fit existing data and forecast nothing, and that borrowing the predictive discipline of machine learning would be a corrective. The argument has force and has drawn thoughtful rebuttals, chiefly that prediction and explanation are different goals rather than competing answers to one goal, so that a field’s choice between them should follow from what it is trying to learn (Shmueli, 2010).

This book’s move is the estimand-first reconciliation it has used throughout: name the target of inference before choosing the method, and let the target dictate both the method and the standard of evidence. Some longitudinal questions are genuinely predictive. Whether a patient will relapse in the next month, which students are at risk of dropping out, when a just-in-time intervention should fire, these are questions about what will happen to a specific person, and they are answered well by a model with high out-of-sample predictive skill whether or not its internals are interpretable. Other questions are structural or causal, about why an outcome occurs or what would change it, and for these a predictive black box is beside the point. The error is not to use machine learning but to mismatch it to the question: to read variable importance as a causal effect, or to dismiss a well-validated predictive model because its coefficients lack a theoretical story. Table 31.1 maps the chapter’s methods onto the goals they serve, and Figure 31.1 draws the two cultures with the estimand-first bridge between them.

Note. Breiman’s two cultures pursue different goals: explanation through an interpretable model assumed correct, and prediction through any function validated out of sample. They are not rivals to be adjudicated but goals to be chosen between. The book’s estimand-first rule resolves the tension: decide whether the question is predictive or structural, then select the method and the standard of evidence that goal requires. The failure mode is the mismatch, reading a predictive model’s internals as explanation or judging an explanatory model by predictive skill.

31.2 Honest Evaluation with Dependent Data

The most important fact in this chapter is that the cross-validation taught for independent observations is biased for repeated measures, and the bias is optimistic. When the rows of a data set are occasions nested within persons, randomly assigning rows to folds places some of a person’s occasions in the training set and others in the test set, so the model is evaluated partly on people it has already learned. Because occasions within a person are correlated, sometimes strongly, the model that has seen a person’s other occasions predicts the held-out ones better than it would predict a genuinely new person, and the cross-validated accuracy it reports overstates the accuracy it would achieve in use. This is leakage, and it is not a subtle bias to be noted and set aside; it is the difference between a model that works in the paper and one that fails in deployment.

The demonstration uses the affect_ema data, where the task is to predict momentary negative affect from momentary predictors and a lagged outcome, and the learner is a random forest. Figure 31.2 shows the result under three resampling schemes. Naive row-wise ten-fold cross-validation, which ignores the nesting and additionally uses a person-level aggregate feature computed on the full sample, reports a test \(R^2\) of \(0.41\). Person-grouped cross-validation, which holds out whole persons so that no test person appears in training, reports \(0.35\). Forward-chaining, which trains on each person’s earlier occasions and tests on their later ones, reports \(0.35\) as well. The naive scheme is optimistic by \(0.06\) in \(R^2\), a seventeen percent relative inflation, and the gap is stable across repeated splits, which places the two estimates more than seven standard errors apart. The persistence baseline, which simply predicts that negative affect equals its previous value, reports \(0.27\), and the comparison it enables is the sobering one: the honest model beats yesterday’s value by only \(0.08\) in \(R^2\). A study that reported the naive figure would claim skill it does not have, and a study that omitted the persistence baseline would claim novelty for a model barely better than doing nothing.

Note. Test \(R^2\) for predicting momentary negative affect in the affect_ema data under three resampling schemes and a baseline. Naive row-wise cross-validation (red) reports \(0.41\); it is optimistic because test persons also appear in training and because a person-aggregate feature was computed on the full sample. Person-grouped and forward-chaining cross-validation (blue), which respect the nesting and the arrow of time, report \(0.35\), the honest estimates. The persistence baseline (grey), predicting the previous value, reports \(0.27\): the honest model’s advantage over doing nothing is modest. Error bars are \(\pm 1.96\) standard errors over repeated splits.

The repair is to match the resampling scheme to the prediction task, and there are two tasks. Predicting a new person’s outcomes, the screening and prognosis case, is evaluated by person-grouped cross-validation, which partitions persons rather than rows into folds so that every test person is entirely unseen during training; its exchangeability argument is that new persons in the test folds stand in for the new persons the model will face in use. Predicting a person’s future from their past, the forecasting and just-in-time case, is evaluated by forward-chaining, also called rolling-origin evaluation, which for each person trains on occasions up to some time and tests on later ones, never letting the future inform the past. Table 31.2 is the selector, and Figure 31.3 draws the two schemes. Two further disciplines complete the picture. Preprocessing, any scaling, imputation, or feature construction, must happen inside the training folds only, because a transformation fit on the whole sample leaks test information into training, exactly the leak the person-aggregate feature introduced above. And when a model’s hyperparameters are tuned, the tuning must itself be cross-validated inside the training folds, a nested cross-validation, because choosing hyperparameters by test-set performance and then reporting that performance is a subtler form of testing on the training data (Varma & Simon, 2006).

Note. Blue cells are training occasions, orange cells test occasions. Person-grouped cross-validation (top) assigns whole persons to the test fold, so no test person is seen in training; it estimates generalization to new persons. Forward-chaining (bottom) trains on each person’s earlier occasions and tests on their later ones, never using the future to predict the past; it estimates within-person forecasting. Naive row-wise splitting, not shown, scatters a person’s occasions across training and test folds and so leaks.

31.3 Trees and Forests for Nested Data

Decision trees partition the predictor space into regions and fit a simple model in each, and their ensembles, random forests, average many trees grown on bootstrap samples with random feature subsets to reduce variance. Applied naively to nested data they mislead in two ways. Because occasions within a person are correlated, a tree can achieve apparent fit by effectively memorizing persons, and its variable importance is biased toward features that happen to track the cluster structure rather than features that carry within-person signal. The repair is to keep the random-effects structure while letting the algorithm search over the fixed part. Mixed-effects trees and the RE-EM procedure (Sela & Simonoff, 2012) alternate between estimating random effects given a tree and growing a tree given the random effects, so the partition explains the fixed differences among subgroups while the random effects absorb the within-person dependence; the generalized linear mixed-model tree (Fokkema et al., 2018) does the same within the mixed-model framework and is the mature tool for detecting subgroups whose fixed effects differ. Mixed-effects forests extend the idea to prediction (Hajjem et al., 2014; Capitaine et al., 2021), though the package ecosystem for longitudinal forests remains less settled than for cross-sectional ones, a maturity caveat worth stating in print.

The worked example asks a subgroup-discovery question of the growth_subgroups data, in which the growth slope is determined by a covariate tree, steep for low-socioeconomic-status urban children, moderate for low-status rural children, and shallow for high-status children, buried among four noise covariates that carry no signal. A growth-parameter tree, which estimates each person’s growth slope from a mixed model and then partitions the slopes on the covariates, recovers the structure: it splits first on socioeconomic status and then on urban residence, ignoring the four noise covariates entirely, and its terminal-node slopes on a held-out validation half, \(1.84\), \(1.10\), and \(0.65\), match the planted truth of \(2.0\), \(1.2\), and \(0.5\) up to the shrinkage that estimating person slopes induces. Figure 31.4 shows the discovered subgroups over the raw trajectories. A permutation-importance analysis of a slope forest confirms the finding, ranking socioeconomic status and urban residence far above the four noise covariates, as Figure 31.5 shows. The structural-equation-model tree of Brandmaier et al. (2013, 2016) generalizes the idea from a slope to a whole latent-growth model, partitioning the sample by covariates that predict differences in any growth parameter, and it inherits the same discipline that Chapter 22 demanded of mixtures: a discovered subgroup is a hypothesis, validated on fresh data, not a discovery certified by the fit that found it.

Note. Mean trajectories of the three subgroups a growth-parameter tree discovered in the growth_subgroups data. The tree split first on socioeconomic status and then on urban residence, the two true moderators, ignoring four noise covariates; the recovered leaf slopes on a held-out validation half (\(1.8\), \(1.1\), \(0.7\)) match the planted truth (\(2.0\), \(1.2\), \(0.5\)) up to the shrinkage of estimated person slopes. A discovered subgroup is a hypothesis to validate, not a finding certified by the search that produced it.

Note. Permutation importance, the increase in prediction error when a predictor is shuffled, from a forest predicting person growth slopes in the growth_subgroups data. Socioeconomic status and urban residence, the two true moderators, dominate; the four noise covariates sit near zero. Importance ranks predictive contribution and is read descriptively: it identifies features the model uses, not causes of the outcome.

31.4 Trajectory Clustering: The Algorithmic Rival

Trajectory clustering groups persons by the shape of their change over time, and it is the algorithmic counterpart to the model-based growth mixtures of Chapter 22. Where a growth mixture posits a probabilistic model, latent classes each with their own growth parameters, and estimates it by maximum likelihood with all the enumeration and fit machinery that entails, a clustering algorithm computes distances between trajectories and groups the near ones with no generative model at all. The k-means-for-longitudinal-data approach (Genolini & Falissard, 2010) treats each person’s sequence as a vector and applies k-means; distance-based variants replace Euclidean distance with a shape-aware distance such as dynamic time warping, which aligns sequences that share a shape but differ in timing before comparing them, appropriate when the timing of features is a nuisance to be removed rather than the signal itself, the same phase-versus-amplitude distinction Chapter 30 drew for registration.

The showdown pits the two philosophies against a known truth. The classes_sim data are generated from three latent growth classes, and Figure 31.6 compares a Gaussian mixture, trajectory k-means, and dynamic-time-warping clustering by how well each recovers the true class labels, measured by the adjusted Rand index. The model-based mixture wins, recovering the classes at an adjusted Rand index of \(0.90\), because its generative model matches the data-generating process; trajectory k-means follows at \(0.79\); and dynamic-time-warping clustering trails at \(0.71\), because its warping flexibility solves a phase-misalignment problem these data do not have and so discards useful amplitude information in the service of an alignment that was never needed. The ordering is not a universal ranking but a lesson about matching method to structure: when the generative model is known and correct, the model-based method is most efficient; when it is unknown or misspecified, the algorithmic methods are more robust; and dynamic time warping earns its keep only when timing misalignment is real. Table 31.4 states the trade-off, and it carries the doctrine of Chapter 22 verbatim: neither a mixture nor a clustering discovers real types, and a recovered grouping is a description to be validated externally, never a taxonomy certified by the algorithm that produced it.

Note. Panel (a): recovery of the three true classes of the classes_sim data, by adjusted Rand index. The Gaussian mixture (\(0.90\)) wins because its generative model matches the data; trajectory k-means (\(0.79\)) follows; dynamic-time-warping clustering (\(0.71\)) trails because its warping addresses a timing misalignment these data do not contain. Panel (b): the true class-mean trajectories (grey) and the recovered mixture-cluster means (dashed) nearly coincide. The ranking is a lesson in matching method to structure, not a universal order.

31.5 Regularization for Longitudinal Problems

Intensive designs create many-predictor problems: dozens of momentary items, their lags, their interactions, and their person-level aggregates can generate more candidate predictors than a classical regression can estimate. Regularization handles this by penalizing coefficient magnitude, shrinking small effects toward zero and, in the lasso, setting some exactly to zero, which performs selection and estimation at once. The penalty is the same shrinkage device that random effects apply, and it is the frequentist cousin of the Bayesian priors of Chapter 17, a connection worth holding onto because it demystifies regularization as disciplined shrinkage rather than a black art. Applied to screening next-occasion negative affect in the affect_ema data from a design matrix of momentary predictors, their lags, and all pairwise interactions, the lasso under person-grouped cross-validation selects a sparse model of seven predictors, retaining momentary stress, positive affect, and the lagged outcome along with a few interactions, and discarding the rest. Figure 31.7 shows the cross-validation path and the one-standard-error rule that chooses the sparse honest model.

Two longitudinal complications deserve care. First, when a predictor is decomposed into its within-person and between-person parts, as Chapter 7 recommended, the two components should be penalized as a pair or with awareness that penalizing them separately can distort the decomposition. Second, and more consequentially, the coefficients that survive selection do not carry the standard errors and p-values of an ordinary regression, because they were chosen by looking at the data, and post-selection inference that ignores this is invalid. The honest options are three: use the regularized model purely for prediction and report only out-of-sample skill; lock the selected model and refit it on a fresh sample where inference is valid; or split the data, selecting on one part and inferring on the other. The pragmatic doctrine is to keep selection and inference in separate compartments, and never to report a lasso coefficient with a naive confidence interval as though the selection had not happened.

Note. The lasso cross-validation path for screening next-occasion negative affect in the affect_ema data, with folds grouped by person so that the selected model is honest for new persons. The error is flat across a wide range of penalties and rises as the penalty grows large; the one-standard-error rule (dashed) chooses a sparse model of seven predictors near the point where sparsity begins to cost accuracy. The selected coefficients are for prediction; inference on them requires a locked model or a fresh sample.

31.6 Idiographic Prediction and the Person-Specific Frontier

The predictive question can be turned inward, from generalizing across persons to forecasting a single person’s future, and here the pooling lesson of Chapter 25 returns in predictive clothing. Two strategies compete. A per-person model is fit to each individual’s own history and used to forecast their future, honoring idiographic variation completely; a pooled model borrows strength across persons, fitting one model to everyone and applying it to each. Which wins depends on how much history each person has, because a per-person model must estimate its parameters from that person’s data alone and is unreliable when the data are few. Figure 31.8 shows the crossover in the affect_ema data. With a short history of six occasions, the per-person model forecasts poorly, a root-mean-squared error of \(0.83\) against the pooled model’s \(0.47\), because six points cannot pin down an individual model; as the history lengthens the per-person error falls, overtaking the pooled model near twenty-four occasions, and with forty occasions the per-person model wins clearly. The lesson is the shrinkage lesson exactly: when data are sparse, pooling toward the population is the better bet, and only when a person has accumulated enough of their own data does an idiographic model earn its independence. The persistence baseline runs beneath both throughout, a reminder that yesterday’s value is a competitor neither model dominates by much.

Note. Panel (a): mean within-person forecast error in the affect_ema data for a per-person model, a pooled model, and persistence, as the per-person training history lengthens. The per-person model is unreliable with a short history and improves as data accumulate, overtaking the pooled model near twenty-four occasions (dashed). Panel (b): with a rich forty-occasion history, the per-person error distribution is lowest. The crossover is the shrinkage lesson of Chapter 25 in predictive form: pool when data are sparse, individualize when they are rich.

Interpreting the models that result, whether pooled forests or per-person fits, requires tools that summarize a black box without overclaiming. Permutation importance, used above, ranks features by how much shuffling them degrades prediction, and it must be computed with the grouping respected, permuting within the resampling structure rather than across it. Partial-dependence and individual-conditional-expectation displays show how the predicted outcome moves as one feature varies with others held fixed or averaged, and Figure 31.9 shows the rising dependence of predicted negative affect on momentary stress, with individual curves revealing heterogeneity around the average. The essential discipline is that these displays are descriptive, not causal: the partial-dependence curve shows how the model’s prediction responds to a feature, which equals the causal effect of that feature only under assumptions the model does not verify and usually cannot. A rising partial-dependence curve for stress means the model predicts higher affect when stress is higher, not that reducing stress would lower affect, and the distance between those two readings is the distance between description and explanation the chapter opened by drawing. The forecasting displays, finally, must always be anchored to a baseline and reported with calibration as well as discrimination, Figure 31.10 placing every scheme’s error against the persistence line, and Table 31.5 listing the metrics and baselines a predictive report requires.

Note. The partial-dependence curve (red) shows how the forest’s predicted negative affect rises with momentary stress, averaged over the other features; the grey individual-conditional-expectation curves show that the dependence varies across observations. These displays describe the model’s behavior, not the world’s causal structure: the curve is the model’s response to a feature, which equals a causal effect only under untested assumptions. Read descriptively, they reveal what the model has learned to use.

Note. Test root-mean-squared error for predicting momentary negative affect under each scheme, with the persistence baseline (dashed) marking the error of predicting the previous value. The naive scheme’s low error is the leakage of Figure 31.2 in error-metric form; the honest schemes sit just below persistence. A predictive claim is only as strong as its margin over the simplest baseline, which for longitudinal outcomes is almost always persistence.

31.7 Reporting and Preregistration

A predictive study earns trust by reporting the decisions that determine whether its accuracy is real, and the reporting standards developed for clinical prediction models transfer directly (Collins et al., 2015). The performance must be reported with uncertainty, from repeated resampling, and against the mandatory baseline, so a reader can see both how accurate the model is and how much it improves on doing nothing. The resampling scheme must be named and justified, and the leakage audit made explicit: how folds respected persons and time, where preprocessing sat relative to the split, and how tuning was nested. The distinction between what was fixed in advance and what the data chose must be preserved, because it is what allows a machine-learning analysis to be preregistered at all. A preregistration can fix the resampling scheme, the metrics, the baselines, the tuning grid, and the feature set, leaving only the fitted model to the data, and an analysis conducted this way occupies the honest exploratory lane the book returns to in Chapter 36: exploratory in that the model is discovered, confirmatory in that the evaluation was specified before the model was seen. Table 31.6 is the checklist, the software box states the package-maturity audit the chapter has promised, and the practice box addresses the compute and reproducibility realities of running these analyses at scale.