Skip to content

The agent's loop: estimand, missing data, multiplicity

In this chapter

  • Read an estimand by its five attributes, and name the strategies for intercurrent events (ICH E9(R1)).
  • Say what LOCF assumes, and why that assumption is weak in Alzheimer’s disease.
  • Explain why two co-primary endpoints need no alpha adjustment, and what they cost.
  • Work one turn of the loop: look, write one block, check it in its place, measure.
Time
45 minutes: the core statistics of the plan
At the end you have
three rounds of the loop: agent score 84 → 90 → 92; the primary analysis states its missing-data handling, and the two primary endpoints have a multiplicity block.

The loop has four steps: look, write one block, check it in its place, measure again. The agent does it; you can do it yourself with the same commands. This chapter works three turns. Each turn answers a question of statistics, so each section starts with the idea.

An estimand is a precise description of the treatment effect that the trial estimates. ICH E9(R1) builds it from five attributes ICH E9(R1) A.3.3:

Attribute The question it answers In EST1
Treatment which treatments are compared placebo, xanomeline low dose, xanomeline high dose
Population in which patients the efficacy population (EFF)
Variable what is measured, when the ADAS-Cog (11) at week 24, as change from baseline
Intercurrent events what happens after randomization that changes the meaning of the measurements, and how each is handled a temporary interruption, a permanent discontinuation: both treatment policy
Population-level summary the number that compares the groups the dose response of the mean change, from an ANCOVA

An intercurrent event happens after the treatment starts and affects the interpretation or the existence of the measurements: the patient stops the drug, takes a rescue medicine, dies ICH E9(R1) A.3.1. The addendum names five strategies for such an event ICH E9(R1) A.3.2:

  • Treatment policy: use the value whether or not the event happened. The effect is that of assigning the drug.
  • Hypothetical: the value as it would have been if the event had not happened (for example, without rescue medicine).
  • Composite: the event becomes part of the variable (for example, a patient who stops counts as a non-responder).
  • While on treatment: the response before the event only.
  • Principal stratum: the effect in the patients in whom the event would not happen.

The agent looks before it writes. sapling show prints the block, the protocol sentence it cites, and every item of the scorecard on it:

Terminal
$ sapling show EST1
estimand EST1 sap.qmd:332 not judged
| ::: {.estimand #EST1 level="primary" population="EFF" analysis="T14_3_1_CHG"}
| **Group mean changes from baseline in the primary efficacy parameters**
|
| [Placebo, or Xanomeline Low Dose, or Xanomeline High Dose]{.treatment}
| [Alzheimer's Disease Assessment Scale - Cognitive Subscale, total of 11 items \[ADAS-Cog (11)\] at Week 24]{.endpoint}
| [Temporary Treatment Interruption]{.intercurrent strategy="treatment policy"}
| [Permanent discontinuation of study treatment]{.intercurrent strategy="treatment policy"}
| []{.cite source="usdm:estimands" words="Estimand EST1: Group mean changes"}
| :::
the protocol (USDM estimands): "Estimand EST1: Group mean changes from baseline in the primary efficacy parameters."
...
OPEN [estimands] EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption
...
OPEN [estimands] EST1 (primary): no sensitivity analysis of the assumptions about the missing data (ICH E9(R1) A.5.2)
...
The whole output (28 lines)
Terminal
$ sapling show EST1
estimand EST1 sap.qmd:332 not judged
| ::: {.estimand #EST1 level="primary" population="EFF" analysis="T14_3_1_CHG"}
| **Group mean changes from baseline in the primary efficacy parameters**
|
| [Placebo, or Xanomeline Low Dose, or Xanomeline High Dose]{.treatment}
| [Alzheimer's Disease Assessment Scale - Cognitive Subscale, total of 11 items \[ADAS-Cog (11)\] at Week 24]{.endpoint}
| [Temporary Treatment Interruption]{.intercurrent strategy="treatment policy"}
| [Permanent discontinuation of study treatment]{.intercurrent strategy="treatment policy"}
| []{.cite source="usdm:estimands" words="Estimand EST1: Group mean changes"}
| :::
the protocol (USDM estimands): "Estimand EST1: Group mean changes from baseline in the primary efficacy parameters."
ok [grounded] estimand EST1 rests on protocol USDM estimands
ok [estimands] EST1: population
ok [estimands] EST1: treatment
ok [estimands] EST1: variable
ok [estimands] EST1: intercurrent events and strategies
ok [estimands] EST1: population-level summary
OPEN [estimands] EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption
how: write ::: {.missingdata #MD_EST1 analysis="T14_3_1_CHG" assumption="MAR" handling="..."} with the justification (EMA 2010; ICH E9(R1) A.5)
OPEN [estimands] EST1 (primary): no sensitivity analysis of the assumptions about the missing data (ICH E9(R1) A.5.2)
how: sapling generate sensitivity (an MNAR analysis: tipping point, or jump to reference), estimand="EST1"; what the protocol does not have is the statistician's addition: propose it in a ::: {.decision}
ok [estimands] EST1: 2 intercurrent event(s), each with its strategy
OPEN [agreed] estimand EST1: not judged yet
how: the statistician reads it (sapling show EST1), then: sapling agree EST1 --by NAME, or sapling revise EST1 "..." --by NAME

Five attributes hold. Two items are OPEN: the estimand does not say how the analysis handles missing values, and it has no sensitivity analysis. Here is the same block as the SAP’s PDF prints it:

The estimand EST1 in the SAP PDF: its title, the analysis T14_3_1_CHG, the level primary, the population EFF, the three treatments, the variable ADAS-Cog (11) at week 24, two intercurrent events with the treatment policy strategy, and the USDM source.
The primary estimand EST1 in the SAP PDF, with its five attributes and its source in the protocol.Source: out/sap/sap.pdf, pages 14 and 15 (sapling build, chapter 11)

The protocol defines the primary endpoint as the last measurement obtained postrandomization. A patient who leaves at week 8 keeps the value of week 8 as the value of week 24. This is LOCF: the last observation carried forward.

LOCF assumes that the score of a patient who stops would have stayed at its last value. In Alzheimer’s disease the ADAS-Cog (11) worsens over time (a higher score is worse). So LOCF freezes a patient who leaves early at a better score than the patient would have had at week 24. The arm with more and earlier dropouts gains from it. The EMA guideline on missing data names this risk: LOCF is acceptable only when its assumption is justified, and it can be anti-conservative in a progressive disease EMA 2010 6.3.1.

This is not a theoretical risk in the pilot. The disposition table of the study (chapter 15) shows how unequal the dropout was:

Table 14.1.2 Subject DispositionPopulation: Intent-to-treat population
StatisticPlacebo (N=86)Xanomeline Low Dose (N=84)Xanomeline High Dose (N=84)Total (N=254)
End of study status
COMPLETED58 (67.4%)25 (29.8%)27 (32.1%)110 (43.3%)
DISCONTINUED28 (32.6%)59 (70.2%)57 (67.9%)144 (56.7%)
Reason for discontinuation
ADVERSE EVENT8 ( 9.3%)44 (52.4%)40 (47.6%)92 (36.2%)
COMPLETED58 (67.4%)25 (29.8%)27 (32.1%)110 (43.3%)
DEATH2 ( 2.3%)1 ( 1.2%)0 ( 0.0%)3 ( 1.2%)
LACK OF EFFICACY3 ( 3.5%)0 ( 0.0%)1 ( 1.2%)4 ( 1.6%)
LOST TO FOLLOW-UP1 ( 1.2%)1 ( 1.2%)0 ( 0.0%)2 ( 0.8%)
PHYSICIAN DECISION1 ( 1.2%)0 ( 0.0%)2 ( 2.4%)3 ( 1.2%)
PROTOCOL VIOLATION2 ( 2.3%)1 ( 1.2%)3 ( 3.6%)6 ( 2.4%)
STUDY TERMINATED BY SPONSOR2 ( 2.3%)2 ( 2.4%)3 ( 3.6%)7 ( 2.8%)
WITHDRAWAL BY SUBJECT9 (10.5%)10 (11.9%)8 ( 9.5%)27 (10.6%)

Program: t14_1_2.R · Output: T14_1_2 · Results: output/ard/t14_1_2.csvSource: out/tlf/listings/t14_1_2.lst (chapter 15)

70.2% of the low dose and 67.9% of the high dose discontinued, most for an adverse event, against 32.6% of placebo.

The agent writes one block that states the handling, its assumption and why the plan keeps it. Then it checks the block in its place, without writing it:

md.qmd
::: {.missingdata #MD_T14_3_1_CHG analysis="T14_3_1_CHG" assumption="MNAR" handling="locf"}
The protocol's 24-week endpoint is the last measurement after randomization: the last observation is carried forward
(LOCF). LOCF assumes that the score of a patient who stops stays at its last value. In Alzheimer's disease the scores
worsen over time, so LOCF can favour the group with earlier dropouts and is not conservative (EMA 2010, 6.3.1). The
plan keeps it as the primary handling, as the protocol defines it, and tests its assumption with a sensitivity analysis.
:::
[]{.cite source="protocol:4.3.2" words="the last measurement obtained postrandomization"}
Terminal
$ sapling block check md.qmd --after T14_3_1
cites 4.3.2: "The primary analysis of ADAS-Cog (11) and CIBIC+ will be the 24-week endpoint, which is defined for each patient and variable as the last measurement obtained postrandomization (prior to protocol defined reduction in dose)."
makes missing_data MD_T14_3_1_CHG (line 398)
BLOCK CHECK PASS: 0 error(s); not written (--write writes it)

The check compiles the block where it will stand, after the table T14_3_1. It found the cited words in section 4.3.2 of the protocol and printed the whole sentence. Only then does the agent write it, and measure:

Terminal
$ sapling block check md.qmd --after T14_3_1 --write
cites 4.3.2: "The primary analysis of ADAS-Cog (11) and CIBIC+ will be the 24-week endpoint, which is defined for each patient and variable as the last measurement obtained postrandomization (prior to protocol defined reduction in dose)."
makes missing_data MD_T14_3_1_CHG (line 398)
BLOCK CHECK PASS: 0 error(s); written to sap.qmd
Terminal
$ sapling round --note "the missing data of the primary analysis"
...
AGENT SCORE 90/100 (+6 since round 1)
Sound 91 #########. 0 error(s), 1 warning(s) of the compiler
Grounded 100 ########## 20 of 20 deciding block(s) cite the protocol or are declared additions
Estimands 88 #########. 7 of 8 estimand check(s) hold
Rigour 91 #########. 0 error(s), 1 warning(s) of the knowledge base's rules open
Complete 74 #######... 40 of 54 checklist item(s) shown in the plan
Feedback 100 ########## no note of the statistician
Executable - not run yet (sapling run): not scored
...
AGAINST ROUND 1: IMPROVED (estimands 75->88, rigour 77->91, complete 70->74); 5 item(s) answered
...
The whole output (24 lines)
Terminal
$ sapling round --note "the missing data of the primary analysis"
sap.qmd (plan 4526401e85f5)
AGENT SCORE 90/100 (+6 since round 1)
Sound 91 #########. 0 error(s), 1 warning(s) of the compiler
Grounded 100 ########## 20 of 20 deciding block(s) cite the protocol or are declared additions
Estimands 88 #########. 7 of 8 estimand check(s) hold
Rigour 91 #########. 0 error(s), 1 warning(s) of the knowledge base's rules open
Complete 74 #######... 40 of 54 checklist item(s) shown in the plan
Feedback 100 ########## no note of the statistician
Executable - not run yet (sapling run): not scored
READINESS NOT READY (the statistician's)
Decisions 22 open decision(s)
Agreed 0 of 81 block(s) agreed
Approved not approved
AGAINST ROUND 1: IMPROVED (estimands 75->88, rigour 77->91, complete 70->74); 5 item(s) answered
NEXT (agent): EST1 (primary): no sensitivity analysis of the assumptions about the missing data (ICH E9(R1) A.5.2)
how: sapling generate sensitivity (an MNAR analysis: tipping point, or jump to reference), estimand="EST1"; what the protocol does not have is the statistician's addition: propose it in a ::: {.decision}
NEXT (statistician): open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47)
SAPLING EVAL: agent score 90/100; NOT READY (22 open decision(s) for the statistician; 81 of 81 block(s) not agreed); saved as round 2

+6, and five items answered by one block: the estimand’s missing-data item, Gamble item 28, the rule on single imputation, and two checklist items. The block records the assumption as MNAR (missing not at random): LOCF does not predict a missing value from the data that were observed, as a MAR method would; it fixes the value by a rule.

A page of the SAP PDF: the analysis of the change from baseline to week 24 with its model, the formula Y equals mu plus beta times baseline plus gamma for the pooled site plus tau times dose plus error, and below it the missing data block MD_T14_3_1_CHG with its text and its protocol source.
The analysis and its missing-data block in the SAP PDF. The model is an ANCOVA of the change with the baseline and the pooled site, and the dose as a number: τ tests a linear dose response.Source: out/sap/sap.pdf, page 17

7.3 Two primary endpoints and the type I error

Section titled “7.3 Two primary endpoints and the type I error”

The protocol names two primary endpoints, ADAS-Cog (11) and CIBIC+, and says that both must reach significance. The type I error is the chance that the trial declares an effect when there is none. With several tests, that chance can grow, so a plan must say how it is controlled ICH E9 2.2.5.

Two rules are possible:

  • Any one endpoint wins (for example ADAS-Cog or CIBIC+). Two chances to win: the type I error grows, and the alpha must be split or tested in a hierarchy.
  • All endpoints must win (ADAS-Cog and CIBIC+). These are co-primary endpoints. The trial wins only if both tests win, so the chance of a false win is not larger than that of one test. No alpha adjustment is needed. The cost is power: with two endpoints, each at 80% power and independent, the chance that both win is about 64%. The FDA guidance also says that raising the alpha of each co-primary test to win back power is not acceptable FDA 2022 III.C.1.

The agent writes the multiplicity block with the procedure co-primary, the two hypotheses, and the protocol sentence:

mult.qmd
::: {.multiplicity #MULT procedure="co-primary" alpha="0.05"}
**The primary hypotheses**
[No linear dose response of the change in the ADAS-Cog (11) at Week 24]{.hypothesis #H1 analysis="T14_3_1_CHG"}
[No linear dose response of the CIBIC+ at Week 24]{.hypothesis #H2 analysis="T14_3_2_V"}
:::
[]{.cite source="protocol:4.3.1" words="Because both of these variables"}
Terminal
$ sapling block check mult.qmd --after T14_3_2 --write
cites 4.3.1: "Because both of these variables must reach statistical significance, an adjustment to the nominal p-values is necessary in order to maintain a .05 Type I error rate for this study."
makes multiplicity MULT (line 431)
BLOCK CHECK PASS: 0 error(s); written to sap.qmd
Terminal
$ sapling round --note "the multiplicity block"
...
AGENT SCORE 92/100 (+2 since round 2)
Sound 91 #########. 0 error(s), 1 warning(s) of the compiler
Grounded 100 ########## 21 of 21 deciding block(s) cite the protocol or are declared additions
Estimands 88 #########. 7 of 8 estimand check(s) hold
Rigour 100 ########## 0 error(s), 0 warning(s) of the knowledge base's rules open
Complete 76 ########.. 41 of 54 checklist item(s) shown in the plan
Feedback 100 ########## no note of the statistician
Executable - not run yet (sapling run): not scored
...
AGAINST ROUND 2: IMPROVED (rigour 91->100, complete 74->76); 2 item(s) answered
...
The whole output (24 lines)
Terminal
$ sapling round --note "the multiplicity block"
sap.qmd (plan 71b9279fb1dc)
AGENT SCORE 92/100 (+2 since round 2)
Sound 91 #########. 0 error(s), 1 warning(s) of the compiler
Grounded 100 ########## 21 of 21 deciding block(s) cite the protocol or are declared additions
Estimands 88 #########. 7 of 8 estimand check(s) hold
Rigour 100 ########## 0 error(s), 0 warning(s) of the knowledge base's rules open
Complete 76 ########.. 41 of 54 checklist item(s) shown in the plan
Feedback 100 ########## no note of the statistician
Executable - not run yet (sapling run): not scored
READINESS NOT READY (the statistician's)
Decisions 22 open decision(s)
Agreed 0 of 82 block(s) agreed
Approved not approved
AGAINST ROUND 2: IMPROVED (rigour 91->100, complete 74->76); 2 item(s) answered
NEXT (agent): EST1 (primary): no sensitivity analysis of the assumptions about the missing data (ICH E9(R1) A.5.2)
how: sapling generate sensitivity (an MNAR analysis: tipping point, or jump to reference), estimand="EST1"; what the protocol does not have is the statistician's addition: propose it in a ::: {.decision}
NEXT (statistician): open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47)
SAPLING EVAL: agent score 92/100; NOT READY (22 open decision(s) for the statistician; 82 of 82 block(s) not agreed); saved as round 3

Rigour reaches 100: no finding of the guideline rules is open.

A round that breaks an item that held before is REGRESSED. It names each item and exits with code 1. The agent corrects or undoes its change before anything else. Every round is saved, so sapling eval history lists them (chapter 10 draws them).

Your turn 7.1 A citation the protocol does not have

Write a missing-data block for the CIBIC+ analysis T14_3_2_V that cites the words a mixed model for repeated measures in section 4.3.2. Check it in its place, without --write. What does the check say, and what does it offer you?

Show the answer
bad.qmd
::: {.missingdata #MD_T14_3_2_V analysis="T14_3_2_V" assumption="MAR" handling="mmrm"}
The CIBIC+ at Week 24 is analysed with a mixed model for repeated measures under MAR.
:::
[]{.cite source="protocol:4.3.2" words="a mixed model for repeated measures"}
Terminal
$ sapling block check bad.qmd --after T14_3_2
line 434: error[cite-words]: "a mixed model for repeated measures" is not in section 4.3.2 of the protocol word for word. fix: copy the words from one of these sentences (sentence=2: … the 24-week endpoint, which is defined for each patient and variable as the last measurement obtained postrandomization (prior to protocol defined reduction in… | sentence=9: For this reason, the primary analysis of the NPI-X will be of the average of all postrandomization NPI-X subscores except for the one obtained at Week 2.)
CITE ERROR protocol:4.3.2|a mixed model for repeated measures: "a mixed model for repeated measures" is not in section 4.3.2 of the protocol word for word.
makes missing_data MD_T14_3_2_V (line 431)
BLOCK CHECK FAIL: 1 error(s); not written (--write writes it)

The check fails with error[cite-words]: the words are not in section 4.3.2. The fix: offers the sentences of that section that are there. Nothing is written, and the command exits with code 1. An agent cannot invent a quote of the protocol.

What you learned

  • An estimand has five attributes; each intercurrent event needs one of five strategies (ICH E9(R1) A.3).
  • Intercurrent events and missing data are different questions; the SAP answers both.
  • LOCF assumes that a score stays at its last value; in a progressive disease it can favour the arm with more dropouts (EMA 2010 6.3.1).
  • Co-primary endpoints need no alpha adjustment, but they cost power (FDA 2022 III.C.1).
  • One turn of the loop: show, one block, block check, --write, round.

Next: 8 · What only you can add →

Found an error or have a question? Write to datastep.studio@gmail.com· The plan is written in Sapling; every R program runs in DataStep Studio for R.

Data: the CDISC pilot study (cdiscpilot01). The protocol as CDISC USDM from the CDISC 360i project (MIT licence); the SDTM data from the PhUSE scripts repository (MIT licence). Public example data: no real patient.

R is a project of the R Foundation. SAS is a registered trademark of SAS Institute Inc. CDISC, SDTM, ADaM and USDM are trademarks of the Clinical Data Interchange Standards Consortium. Claude is a trademark of Anthropic. This book is not affiliated with or endorsed by any of them.

© 2026 DataStep Labs. All rights reserved.