Dynamic Factor Analysis and State-Space Models

26.1 One Form to Hold Them All

The state-space form is two equations. The observation equation, \(\mathbf{y}_t = \mathbf{Z}\,\boldsymbol{\alpha}_t + \boldsymbol{\varepsilon}_t\) with \(\boldsymbol{\varepsilon}_t \sim N(\mathbf{0}, \mathbf{H})\), says that what is measured at occasion \(t\) is a loading matrix \(\mathbf{Z}\) times a latent state \(\boldsymbol{\alpha}_t\), plus measurement error \(\boldsymbol{\varepsilon}_t\). The state equation, \(\boldsymbol{\alpha}_t = \mathbf{T}\,\boldsymbol{\alpha}_{t-1} + \boldsymbol{\eta}_t\) with \(\boldsymbol{\eta}_t \sim N(\mathbf{0}, \mathbf{Q})\), says that the latent state evolves according to a transition matrix \(\mathbf{T}\), driven by process noise \(\boldsymbol{\eta}_t\). The two noise terms are not the same thing, and keeping them apart is the whole point. The measurement error \(\boldsymbol{\varepsilon}_t\) is assessment noise, the unreliability of the instrument, the part of a mood rating that reflects the vagueness of the item rather than the person’s actual state. The process noise \(\boldsymbol{\eta}_t\) is the genuine shock, the true change in the person’s state from one occasion to the next. Every previous chapter that regressed an observed score on its own lag silently fused these two, and this chapter separates them. Figure 26.1 draws the two equations with their psychological glosses.

Note. The state equation lets a latent state \(\alpha_t\) evolve through a transition matrix \(\mathbf{T}\), disturbed by process noise (true change). The observation equation measures the state through a loading matrix \(\mathbf{Z}\), corrupted by measurement error (assessment noise). Separating the two noises is the form’s central psychometric contribution. The diagram is reused in Chapters 27 and 32.

The immediate payoff of the separation is a fact that reframes Chapter 24. An autoregression of order one observed with measurement error is not an autoregression at all; algebraically it is an ARMA(1,1), and its naive lag-one autocorrelation understates the true carryover in proportion to the unreliability of the measure. The mechanism is exactly the attenuation that measurement error inflicts on any correlation, applied here to the correlation of a series with its own past. Figure 26.2 makes this concrete on kalman_n1, a single-subject series generated from a latent AR(1) with true carryover \(0.60\) observed with error at a reliability of \(0.65\). The left panel shows the naive autoregression falling steadily as reliability worsens, reaching well below the true value; on this series the naive lag-one autocorrelation is \(0.46\), badly short of the generating \(0.60\). The right panel shows the repair and its cost together: a Kalman-based maximum-likelihood fit that models the measurement error separately recovers a carryover of \(0.61\), and across a hundred simulated series it is unbiased on average, but its per-series estimate is noisy, because separating two variances from one short series is a demanding request. The separation succeeds in expectation and must be reported with its uncertainty.

Note. Left: the naive autoregression of an error-laden series falls below the true carryover (\(0.60\)) as reliability worsens. Right: sampling distributions across 100 single series; the Kalman estimate that models measurement error separately is centered near the truth while the naive estimate is attenuated toward zero. The separation is unbiased on average but imprecise from one series.

The same two equations, with different matrices, become the familiar models, and filling those matrices is the competence this section builds. Figure 26.3 shows five cards. An AR(1) with measurement error has a scalar state, \(\mathbf{T} = \phi\), \(\mathbf{Z} = 1\), process variance \(\mathbf{Q} = \sigma^2_\eta\), and measurement variance \(\mathbf{H} = \sigma^2_\varepsilon\). A local linear trend, the structural time-series model of a level that drifts with a slowly changing slope, has a two-dimensional state of level and slope with a transition matrix that carries the slope into the level. A VAR(1) has a vector state and a full transition matrix, the model of Chapter 25 with \(\mathbf{H} = \mathbf{0}\) when measured without error or \(\mathbf{H} \neq \mathbf{0}\) when not. A dynamic factor model has a latent factor state that evolves and a loading matrix that measures it through several indicators. And the latent growth curve of Chapter 19 is a state-space model with a state of intercept and slope, a transition matrix that holds them constant, and a time-varying loading matrix that reads the slope onto successive occasions, which unifies Parts V and VI under one form. Table 26.1 names each matrix with its dimension, its psychological meaning, and its structural-equation or multilevel analogue, and Table 26.2 states what each of the five models puts in each matrix.

Note. Each familiar model is the state-space form with particular matrices. The autoregression, the trend model, the vector autoregression, the dynamic factor model, and the latent growth curve differ only in what the state contains, how the transition matrix moves it, and how the loading matrix reads it. The latent growth loading is time-varying, which is how a static growth model becomes a state-space model.

26.2 The Kalman Engine

The Kalman filter is a single recursion that walks forward through the series, and at each occasion it does two things. It predicts: given the best estimate of the state at the previous occasion, the transition matrix carries it forward to a prediction of the state now, and the uncertainty grows by the process noise. It updates: the new observation arrives, its discrepancy from what the prediction implied is the innovation, and the state estimate is corrected toward the observation by an amount called the Kalman gain. Figure 26.4 draws the loop. The gain is the heart of the matter and it is a reliability weight in exactly the sense of Chapter 13’s shrinkage. When the measurement is precise relative to the state uncertainty, the gain is large and the estimate moves most of the way to the observation; when the measurement is noisy, the gain is small and the estimate stays close to the model’s prediction. The filter is optimally trading a prior forecast against a noisy datum, precision against precision, which is the same weighted-average logic that shrinks a person’s estimate toward a group mean, now operating in time.

Note. At each occasion the filter predicts the state from the transition matrix (uncertainty growing by the process noise), then updates toward the new observation by the Kalman gain, a reliability weight that is large when the measurement is precise and small when it is noisy. The likelihood, the missing-data handling, and the standardized residuals fall out of the same pass.

Three things fall out of that single forward pass, and together they are why the state-space form is worth its machinery. The likelihood comes free through the prediction-error decomposition: each innovation, standardized by its own variance, contributes a Gaussian term, and their sum is the exact log-likelihood, so maximum-likelihood estimation of the transition, loading, and variance parameters is an optimization over repeated filter runs. Missing data are handled exactly and without imputation: when an occasion is unobserved, the update step is simply skipped, the prediction carries forward, and the state uncertainty grows until the next observation arrives, which is the principled repair of the naive interpolation that Chapter 24 showed inflates autocorrelation. And the standardized one-step-ahead innovations are the residuals for diagnosis, white and normal when the model is right. The foundations box states the likelihood decomposition and the AR-plus-error algebra formally.

The Kalman arc on kalman_n1 demonstrates the separation. Fitting the AR-plus-error model by maximizing the filter likelihood recovers a carryover of \(0.61\) against the generating \(0.60\), and the smoother then produces the best retrospective estimate of the latent state at each occasion. Figure 26.5 shows the filtered and smoothed latent paths with their uncertainty ribbons beneath the noisy observations. The filtered estimate uses only the past and is available in real time; the smoothed estimate uses the whole series and is tighter and closer to the true latent state, the retrospective best guess. The choice between them is a choice of estimand, set out in Table 26.3: filtering is for monitoring as data arrive, the clinical use that Chapter 33 develops for just-in-time interventions, while smoothing is for the historical reconstruction that a completed study wants.

Note. The filtered (past only) and smoothed (whole series) estimates of the latent state on kalman_n1, with the smoothed uncertainty ribbon, against the noisy observations and the true latent path. Both recover the signal beneath the assessment noise; the smoother is tighter and tracks the truth more closely.

The missing-data repair is the payoff Chapter 24 promised. Masking a stretch of kalman_n1 and running the filter, the state estimate through the gap is not a straight line but a curve that reverts toward the process mean while its uncertainty ribbon widens to reflect the absence of data, and closes again when observations resume. Figure 26.6 contrasts this with linear interpolation, which draws a confident straight line through the gap and thereby claims a certainty it does not have. The consequences are measurable: the Kalman smoother reconstructs the latent trajectory in the gap with a root-mean-square error of \(1.43\) against the true latent state, while interpolation manages only \(2.04\), and the carryover estimated on the gapped series by the Kalman method, \(0.57\), stays close to the full-data value, whereas interpolating and then estimating naively inflates the observed autocorrelation. The filter treats a missing occasion as what it is, an absence of information, rather than as a value to invent. The run-it-in-R panel carries the whole arc.

Note. Through a masked stretch of kalman_n1, the Kalman smoother’s uncertainty ribbon widens and its estimate reverts toward the mean, tracking the true latent state (RMSE \(1.43\)); linear interpolation draws a confident straight line that does not (RMSE \(2.04\)). The filter represents missingness as absence of information, not as an invented value.

# The Kalman arc on kalman_n1: separate measurement error from process noise
d <- readRDS("Examples/data/kalman_n1.rds")            # y (observed), eta_true (latent)
yc <- d$y - mean(d$y)
# maximize the filter likelihood over (phi, process var q, measurement var r)
negll <- function(par) {                               # kalman_filter() hand-rolled in ch26_analysis
  phi <- tanh(par[1]); q <- exp(par[2]); r <- exp(par[3])
  -kalman_filter(matrix(yc), matrix(phi), matrix(1), matrix(q), matrix(r),
                 a1 = 0, P1 = matrix(q/(1-phi^2)))$loglik
}
fit <- optim(c(atanh(.3), 0, 0), negll, method = "BFGS")
tanh(fit$par[1])                                       # phi ~ 0.61 (naive lag-1 autocorr = 0.46)
# smoothed latent state + widening uncertainty across a masked gap:
kf <- kalman_filter(matrix(d$y_gap - mean(d$y_gap, na.rm=TRUE)), ...)  # NA occasions skipped
ks <- kalman_smoother(kf, matrix(tanh(fit$par[1])))

26.3 Dynamic Factor Models

The dynamic factor model is the state-space form with a latent factor for its state, and it is the repair of the P-technique that Chapter 24 left broken. P-technique factor-analyzed one person’s items across occasions but treated the occasions as independent, discarding the serial dependence that is the point of a time series. The dynamic factor model keeps the factor structure, a small number of latent factors that load onto the observed items, and adds dynamics, letting the factors evolve as a vector autoregression, so that the latent process both is measured, through the loading matrix, and carries itself forward, through the transition matrix. The result is a person-specific measurement model with person-specific dynamics, idiographic measurement realized. Two canonical forms exist and the distinction is worth stating, as Figure 26.7 draws it. In the process-factor form, the tradition of Molenaar, the latent factor follows the autoregression and the measurement is contemporaneous and clean, so the dynamics live entirely in the factor. In the direct-autoregressive form, the autoregression is written on the factor scores with lagged loadings, distributing the memory across the measurement. The forms can be made equivalent under conditions, and Table 26.4 states when each is preferred and what each requires for identification.

Note. Process-factor form (left): the latent factor follows the autoregression and the loadings measure the current factor cleanly, so all dynamics reside in the factor. Direct-autoregressive form (right): the memory is carried by lagged loadings from earlier factor scores onto current indicators. The forms coincide under conditions; the process-factor form is the more common in psychology.

That the model recovers a known structure is shown on dfm_sim, a two-factor series with a known loading matrix and a known factor transition, fit here by a hand-written expectation-maximization algorithm whose expectation step is the Kalman smoother and whose maximization step updates the loadings, the transition, and the variances in closed form. Figure 26.8 shows the recovery: the loadings return close to their generating values, the factor transition recovers the autoregressive diagonal and the cross-lag from the second factor into the first, and the smoothed factor scores correlate with the true factors at \(0.94\) and \(0.91\). The estimator reconstructs a latent dynamic structure from noisy multivariate indicators, the validation that licenses its use where truth is unknown.

Note. Estimated versus generating loadings (left) and factor transition matrix (right, estimated with the true value in parentheses) for a two-factor dynamic factor model fit to dfm_sim by Kalman-EM. The loadings and the factor VAR are recovered, and the smoothed factor scores correlate with the true factors at \(0.94\) and \(0.91\).

The idiographic promise is realized on real items. Fitting a one-factor dynamic factor model to the four negative-affect items of two affect_ema persons, with the factor free to follow its own autoregression, returns two genuinely different pictures, shown in Figure 26.9. For the first person the negative-affect factor is coherent and persistent: all four items relate substantially to it, feeling nervous most strongly, and the factor carries a moderate inertia of \(0.41\), so their negative affect lingers. For the second person the factor has almost no carryover, an inertia of \(0.06\), and a different indicator profile: feeling tense is its strongest sign while feeling nervous is nearly unrelated to it. The same four items assemble into different personal structures with different dynamics, which is the nonergodicity of Chapter 24 made concrete at the level of measurement itself, and it is precisely what a single between-person factor analysis, imposing one loading pattern and no dynamics on everyone, cannot represent. Multi-person strategies follow two routes: the replicated single-subject approach fits each person separately and summarizes the distribution of their structures and dynamics, while the state-of-the-art multilevel dynamic factor model pools them, which is the measurement layer of the DSEM of Chapter 25, the same idea reached from the factor-analytic side.

Note. One-factor dynamic factor models of four negative-affect items for two affect_ema persons, showing each item’s correlation with the person’s factor. Person 3’s factor is coherent and persistent (inertia \(0.41\)); Person 17’s has near-zero inertia (\(0.06\)) and a different indicator profile. Person-specific measurement and dynamics, which a single between-person model cannot capture.

26.4 Diagnostics and Selection

A fitted state-space model is judged, like any time-series model, by whether it has left white noise behind, and the object of scrutiny is the standardized one-step-ahead innovation, the filter’s residual divided by its own predicted standard deviation. If the model is correctly specified these standardized innovations are independent and standard normal, so three checks apply. Their autocorrelation should be negligible, confirmed by eye and by a Ljung-Box test, because remaining serial structure means the dynamics are misspecified. Their distribution should be normal, checked by a quantile plot and a normality test, because heavy tails signal outliers or a non-Gaussian process. And individual large values flag outliers, which in a psychological series are often not nuisances to be trimmed but substantive events, a stressor, a crisis, a change of regime, to be understood rather than discarded, the pointer to Chapter 32. Figure 26.10 shows the panel for the AR-plus-error fit on kalman_n1: the standardized innovations scatter within their bands, their autocorrelation is negligible with a Ljung-Box \(p\) of \(0.93\), and their quantile plot is straight with a normality \(p\) of \(0.23\), so the model has whitened the series and is retained.

Note. Standardized one-step-ahead innovations from the AR-plus-error fit on kalman_n1: their series within \(\pm 2\) bands (left), their autocorrelation with the Ljung-Box result (center), and their normal quantile plot (right). White, normal, outlier-free residuals indicate an adequate model.

Selection among specifications follows the parsimony doctrine that this book has urged throughout, sharpened by the reality of psychological series lengths. Nested models, an AR-plus-error against a plain AR, or a one-factor against a two-factor dynamic model, are compared by a likelihood-ratio test using the prediction-error likelihood; non-nested ones by an information criterion. The temptation to enrich the state, to add a second factor, a higher lag, a time-varying parameter, must be resisted at the series lengths psychology collects, because a state-space model with more parameters than the data can identify will converge to a boundary, report a variance component pinned at zero, or fit the sample and fail to replicate, the same Heywood-case and over-parameterization hazards that Chapter 13 raised for variance components. The pitfalls box collects the specific ways these models mislead.

26.5 Software and When the Machinery Pays

The state-space form is served by several R packages whose strengths differ, and Figure 26.11 and Table 26.5 lay out the landscape. The KFAS package is fast and exact for linear Gaussian models and their exponential-family extensions, the workhorse for the filtering and smoothing of this chapter. The MARSS package specializes in multivariate autoregressive state-space models with an accessible interface for ecological and psychological time series. The dynr package, whose model-cooking conventions recur in Chapter 32, adds regime-switching and nonlinear dynamics, the readiness for the change-point and oscillator models to come. And OpenMx defines state-space models in a structural-equation environment with definition variables, the natural home for readers fluent in SEM. Mplus, through its dynamic structural equation modeling, is the adjacent production tool whose distinctive addition is multilevel pooling across persons; the distinction worth holding is that DSEM adds the multilevel layer while the state-space packages add structural freedom, the ability to specify custom transition and measurement structures that a fixed DSEM template does not expose. None of these packages was used for the analyses here, which were built from a hand-written filter, smoother, and EM algorithm so that every object could be checked against a known truth; readers with production needs should reach for the packages, and readers who want to understand what the packages do will find the mechanisms in the shipped scripts.

Note. A capability matrix for the main state-space tools. Strengths differ: KFAS for exact speed, dynr for regime-switching and nonlinearity, OpenMx for SEM-style definition, and Mplus DSEM for multilevel pooling. The choice follows what the analysis needs.

When the state-space machinery earns its overhead is the practical question, and the answer is specific. It pays when measurement error must be separated from process noise, because no observed-score model can do that; when data are missing and the missingness must be handled exactly rather than imputed; when the target is a latent multivariate process, a dynamic factor structure that no single-indicator model reaches; and when the structure is custom, a transition or measurement matrix that a packaged model does not offer. It does not pay when a multilevel VAR of Chapter 25 answers the question with less assembly and the measurement error is negligible, in which case the simpler tool is the honest choice. The continuous-time models of the next chapter are the same state-space architecture with the transition governed by a differential equation rather than a discrete step, which is how the form accommodates the unequal intervals that Chapter 25 flagged, and the regime-switching models of Chapter 32 let the transition matrix itself change, which is how the form accommodates the change points this chapter’s outliers hinted at.

Writing up a state-space or dynamic factor analysis extends the reporting of the previous chapters with the decisions specific to the form. The report should state the state-space model fit, what the state contains and how the transition and measurement matrices are specified, and in particular whether measurement error was modeled or fixed to zero, because that single choice governs whether the dynamics are attenuated. It should give the estimated dynamics with their psychological reading and their uncertainty, report how missing data were handled, and present the convergence and residual diagnostics, the standardized-innovation checks, that certify the fit. For a dynamic factor model it should state the identification, the anchor or scale constraint and the lag structure, and for an idiographic analysis it should be explicit that the structure and dynamics are person-specific and not assumed to generalize. A model sentence reads: “A one-factor dynamic factor model with a latent AR(1) factor and freely estimated measurement-error variances was fit to the four negative-affect items by Kalman-EM; the factor’s inertia was \(0.41\) for Person 3 and \(0.06\) for Person 17, and the two persons’ loading profiles differed, indicating person-specific measurement and dynamics; standardized innovations were white and approximately normal.”