The Complete Workflow: Method Selection, Reproducibility, and Reporting

36.1 The Master Decision System

The routing that Chapter 1 introduced as a single figure can now be stated as a system with three inputs and three outputs. The inputs are the question family, one of the five the book has used throughout, the data structure, described by the number of occasions per person, the number of persons, the regularity of the timing, and the type of the outcome, and the assumption budget, the set of assumptions the design and data let you defend, about the missingness mechanism, the invariance of measurement, the stationarity of dynamics, and the interval meaning of the scale. The outputs are the primary analysis that the inputs imply, the mandatory sensitivity analyses that the primary’s assumptions require, and the ceiling on the claims the design can support. Figure 36.1 is the full map, the counterpart of Figure 1.5, now populated with every method the book has developed and closed at the bottom by the workflow spine that this chapter formalizes. Table 36.1 is the companion that most students need first, because the hardest step is not choosing a method once the question is named but phrasing the question so that its estimand is unambiguous.

Note. Enter at the top by naming the estimand, follow the data structure to its category, and read the leaf for the methods that answer each question family, now including the flexible, dynamic, and nonlinear tools developed after Chapter 1. The purple band lists the cross-cutting choices; the green band adds the workflow spine this chapter formalizes. This is the same map Figure 1.5 drew as a promise, revisited as an audit now that every method exists.

A decision system guides; it does not absolve. When the design is novel, when several routes are equally defensible, or when the assumptions cannot be ranked, the system does not fail so much as hand the problem back with its structure exposed, and the honest response is not paralysis but a sensitivity portfolio that runs the defensible routes and reports their agreement or divergence. The three applications chapters showed the system working on real questions: a clinical trial routed through growth, dropout, and joint modeling; a developmental cascade routed through the panel-model family with its estimand discipline; a dyadic diary routed through the multilevel actor-partner model. The routings are worked there; the point here is that each began by naming the estimand and let the map do the rest.

36.2 The Reproducible Longitudinal Project

A result that cannot be rebuilt is a claim, not a finding, and longitudinal projects are especially fragile because they chain many stages, cleaning, deriving, fitting, and drawing, across data that arrive messy and evolve. The reproducible project rests on a few disciplines. The raw data are immutable, read but never written, so that every derived dataset can be traced to an untouched origin. The package environment is locked, so that the versions that produced a result are recorded and restorable, because a longitudinal model fit under one version of a mixed-model library can differ from the next. The pipeline is managed, its stages declared with their dependencies so that changing one input reruns exactly what depends on it and nothing else, which is both an efficiency and a form of documentation, since the dependency graph is a map of the analysis. Every stochastic step, every Markov chain, bootstrap, simulation, and imputation, draws from a registered seed, so that randomness is reproducible rather than merely plausible. And the whole is archived to a repository with a codebook, so that a reader years later can rebuild it. Figure 36.2 shows the anatomy, Figure 36.3 shows a pipeline graph for the exemplar, and Table 36.2 is the seed registry every project should keep.

Note. Raw data are immutable inputs; functions in a code directory and a locked package environment build the derived data, which feed model fits, which feed figures and the report, which are archived to a public repository. The arrows are the pipeline’s dependencies: changing an input rebuilds exactly what depends on it. The directory conventions follow the lab standards of versioned filenames, script headers, and checkpointed long computations.

Note. Each node is a build target and each arrow a dependency. The primary and multiverse fits both depend on the derived data and are independent of each other, so a change to the multiverse code reruns only its branch. The graph is generated by the pipeline tool rather than drawn by hand, so it cannot drift from the actual analysis, which is its value as documentation.

36.3 Reporting Standards, Harmonized

The reporting checklists that accumulated through the book, for mixed models, growth models, invariance, mixtures, dynamic models, networks, survival, and flexible and machine-learning methods, share a common spine, and Table 36.3 states it once with the per-family additions noted. The spine has seven elements: a description of the data structure, missingness, and compliance; a specification of the model as equations or unambiguous prose with the syntax in a supplement; a statement of the estimation method, software, versions, settings, and convergence diagnostics; the inference procedure with its degrees of freedom or diagnostics; the estimands and their effect sizes; the sensitivity results; and the visualization obligations. The per-family deltas are specific and are what a methods reviewer now checks: a growth model must report its time coding and its random-effect structure, a dynamic model its stationarity handling and its detrending, a mixture model its enumeration criteria and its class-size caveats, a survival model its proportional-hazards check, a flexible model its smoothing basis and its concurvity. Journal reality intrudes here, because word limits force material to supplements, and the discipline is to move detail to the supplement without hiding the result, so that a reader of the main text learns what was found and a reader of the supplement can rebuild it.

36.4 The Sensitivity Doctrine, Formalized

Every methods chapter insisted on sensitivity analysis, and the insistence can now be formalized as a registry and a protocol. The fork registry, Figure 36.4 and Table 36.4, catalogues the analytic forks that recur by model family, the centering variants, the time codings, the random-structure alternatives, the missingness models, the detrending and lag choices, the enumeration and tuning decisions, and marks each as must-run or context-dependent. The protocol is the multiverse: preregister the fork set, run it systematically over the pipeline’s grid, and report the result as a specification-summary display that shows every estimate with the analytic choices behind it, so that the pattern of divergence becomes visible. The crucial interpretive move is that the forks which move the conclusion are the finding, because a conclusion robust to every defensible choice is strong precisely to the degree that the multiverse could have overturned it and did not, and a conclusion that flips on a centering decision is a conclusion about the centering. What separates a multiverse from p-hacking is disclosure and pre-specification: the multiverse reports all specifications, the p-hack reports the flattering one.

The exemplar makes this concrete. The question is a within-person one, whether a person’s negative affect rises on occasions when that person’s stress is higher than usual, estimated on the intensive-longitudinal affect_ema records, and the primary analysis, a within-person-centered random-slope multilevel model, estimates the reactivity at 0.35 with a standard error of 0.02. The micro-multiverse re-estimates it across a grid of twenty-four defensible specifications, crossing the centering, the random structure, the lag, and the covariate set, and Figure 36.5 shows the result: every one of the twenty-four specifications returns a positive reactivity, in a tight band from 0.31 to 0.36, so the conclusion that this reactivity is real and positive is robust to every fork, and the fork that moves the magnitude most is the covariate set, controlling for concurrent positive affect pulling the estimate down slightly. The honest paragraph writes itself from the display: the reactivity is positive across all defensible specifications, its magnitude is stable near a third of a scale point of negative affect per unit of within-person stress, and no analytic choice overturns the conclusion, which is the strongest thing a single study can say and is sayable only because the multiverse was run and reported rather than a single specification chosen.

Note. Rows are model families, columns are recurring analytic forks, and cells mark a fork as must-run (blue), context-dependent (orange), or not applicable (grey) for that family. A growth model must run its centering, time coding, random structure, and missingness forks; a survival model must run its time coding, missingness, and interval forks. The registry turns the book’s scattered sensitivity advice into a checklist that a preregistration can commit to in advance.

Note. Panel (a): the within-person stress reactivity estimated under each of twenty-four defensible specifications, sorted, with confidence intervals; all are positive and lie in a narrow band around the primary estimate (dashed). Panel (b): the analytic choices behind each specification, one row per fork level. The specifications with the smallest estimates are those that control for concurrent positive affect, identifying the covariate set as the fork that moves the magnitude; no fork changes the sign or the qualitative conclusion. That robustness, visible only because the whole grid was run, is the finding.

36.5 Preregistration for Longitudinal Research

Preregistration buys a specific thing, the separation of confirmatory from exploratory analysis, and it does not buy freedom from thought or a guarantee of truth. For longitudinal and experience-sampling research it has particular shapes, and Table 36.5 maps what can be bound in advance against what must remain data-contingent. A great deal is bindable even when model-building is genuinely iterative: the estimand, the primary specification, the mandatory fork set, the inclusion and compliance rules, and the decision tree that governs how the model will be built are all specifiable before the data are seen, and the honest middle path is to preregister the decision tree, the rule for how choices will be made, rather than a single fixed endpoint that real data will force you to abandon. Secondary and archival data, on which much longitudinal work rests, remain partly preregisterable: the variables not yet examined, the analyses to be run on a held-out split, and the sensitivity set can be committed even when the data already exist, because preregistration disciplines the analyst, not the data. Deviations are inevitable and are not failures; what matters is that they are logged and reported, so that a reader can see where the analysis departed from its plan and judge the departure. The experience-sampling preregistration templates that the field has developed, in the lineage of Kirtley and colleagues, walk through the sections a dense within-person design requires, and a filled template for the clinical exemplar of Chapter 33 would specify its sampling scheme, its compliance handling, its primary within-person estimand, and its sensitivity set before a single beep were analyzed.

36.6 Writing and Archiving

Writing longitudinal results well begins with the estimand: a topic sentence that names the target quantity, “the within-person reactivity of negative affect to stress was positive and stable across specifications,” orients a reader as a sentence about significance does not. The drafting workflow is figures-first, building the display that shows the result before the prose that describes it, because a result that cannot be shown usually cannot be trusted, and the visualization obligations of the reporting checklist are met by drawing the data, not only the model. Uncertainty language is a discipline of its own, and Table 36.6 pairs the phrases to avoid with their honest replacements, because the difference between “stress causes negative affect” and “on higher-stress occasions, negative affect was higher within person” is the difference between a claim the design cannot support and one it can. Structuring a multi-model results section means marking its lanes, a primary lane, a sensitivity lane, and an exploratory lane, with headers that enforce the epistemics so that a reader never mistakes an exploratory dynamics analysis for a confirmatory one.

Archiving closes the loop the first chapter opened. A public repository for a longitudinal study has an anatomy, Figure 36.6, of raw and derived data, code, a codebook, the preregistration, and the manuscript, tiered so that what can be open is open and what must be protected is access-controlled. The protection matters most for intensive longitudinal data, whose dense behavioral traces are re-identifiable in ways that sparse survey data are not: a sequence of timestamps and locations can single out a person even when names are removed, and Table 36.7 lists the techniques, timestamp coarsening, geodata removal, and the k-anonymity intuition that no record should be unique on its quasi-identifiers, with what each protects against and what it costs. When the raw traces cannot be safely opened, a synthetic companion generated from the fitted model, the very device this book has used for every worked example, lets a reader rebuild the analysis without exposing a participant, which is a fitting closing symmetry: the simulated datasets that taught the methods are also the template for sharing real ones safely.

Note. A tiered repository: derived data, code, codebook, preregistration, and manuscript are open, while raw intensive data, which can re-identify participants, are access-controlled. When even a tiered release is unsafe, a synthetic dataset generated from the fitted model lets a reader rebuild the analysis without exposing anyone, the same device this book used to teach every method.

36.7 The End-to-End Exemplar

The exemplar can now be read as a whole, a compact study executed through the entire system. The question, phrased to fix its estimand, is whether negative affect rises within a person on higher-stress occasions, which the estimand-phrasing guide routes to a within-person multilevel model. The preregistration, filed before analysis, names that estimand, the within-person-centered random-slope specification as primary, the twenty-four-cell fork grid as the mandatory sensitivity set, and a decision rule for the random structure. The pipeline builds the derived data from the immutable raw beeps, fits the primary model and the multiverse as independent branches, and draws the displays, every stochastic step seeded from the registry. The primary analysis returns a reactivity of 0.35, and Figure 36.7 shows the result the way it should be shown, not as a single number but as a distribution of person-specific reactivities around the pooled estimate, positive on average and varying across persons, so that the between-person heterogeneity in a within-person effect is visible rather than hidden. The sensitivity portfolio, the specification curve of Figure 36.5, confirms the estimate is positive across all twenty-four specifications and identifies the covariate set as the only fork that moves it appreciably. The reporting audit against the harmonized checklist confirms the data structure, specification, estimation versions, inference, estimand, sensitivity, and visualization are all present. And the archive deposits the derived data, the code, the codebook, the preregistration, and a synthetic companion to a tiered repository. Each move traces to its chapter, and the whole is a study a stranger could rebuild.

Note. The person-specific stress reactivities of negative affect, with the pooled estimate marked. The effect is positive for nearly every person and varies in magnitude across persons, the between-person heterogeneity in a within-person effect that a single pooled number would conceal. Showing the distribution rather than the point is the visualization obligation the reporting checklist imposes and the estimand-first writing the last section urged.

The book set out to make change analyzable, and it end where it began, at the map. Chapter 1 drew that map as a promise to a reader who did not yet know which method to use; this chapter has walked it as an audit, with every method built, every checklist harmonized, every doctrine formalized, and the whole shown working on one honest example. The claim the two bookend chapters make together is that analyzing change is not a catalogue of techniques to be memorized but a discipline to be practiced: name the estimand, choose the method the question implies, run the sensitivities its assumptions demand, report what was done so that a stranger can see it, and archive it so that a stranger can rebuild it. The reader who has come this far can do this, and that was the point.