Skip to content

What only you can add: a tipping-point analysis

In this chapter

  • Say what a sensitivity analysis is for, and which assumption it must change (ICH E9(R1) A.5.2).
  • Read a tipping-point analysis: the shift delta, and what the tipping point means.
  • Tell what the protocol says from what only the SAP adds, and declare an addition.
Time
25 minutes
At the end you have
round 4: agent score 95, Estimands 100; the plan has a tipping-point analysis that you declared as an addition to the protocol.

8.1 The idea: test the assumption, not the method

Section titled “8.1 The idea: test the assumption, not the method”

Chapter 7 kept LOCF as the primary handling of the missing values, and said that its assumption is weak. ICH E9(R1) asks for sensitivity analyses: analyses of the same estimand under other assumptions, to see whether the conclusion holds ICH E9(R1) A.5.2.1. A sensitivity analysis changes the assumptions in a structured way, one at a time ICH E9(R1) A.5.2.2. Analyses that make similar assumptions to the primary one teach little EMA 2010 7.

The protocol of the pilot plans LOCF and an observed-case analysis. Both assume, in different ways, that the patients who left are like the ones who stayed. Neither tests what happens if they were worse.

A tipping-point analysis asks: how much worse must the missing values be before the conclusion changes?

  1. Impute the missing values of the active groups under MAR (missing at random: like similar patients who stayed), many times (here 100 imputations).
  2. Make them worse by a shift delta: add delta points to each imputed ADAS-Cog (11) value of the active groups. A higher score is worse.
  3. Analyse each data set and combine the results. Increase delta step by step.
  4. The tipping point is the smallest delta at which the primary test is no longer significant.

You then judge the tipping point clinically. If the conclusion changes only when the patients who left are, say, 8 points worse than similar patients who stayed, and such a decline is not plausible, the conclusion is robust. If 1 point is enough, it is fragile. You plan this before the unblinding, when you do not know the result yet.

The next item of the loop asks for a sensitivity analysis under MNAR. The agent writes it with the conventional block of Sapling:

Terminal
$ sapling generate sensitivity
insert SENS1 in "Additional analyses" (sap.qmd)
check 0 error(s), 2 warning(s): SAPLING CHECK PASS
$ sapling show SENS1
sensitivity SENS1 sap.qmd:468 not judged
| ::: {.sensitivity #SENS1 estimand="EST1" method="tipping-point" assumption="MNAR" delta="0 to 6 by 0.5"
| imputations="100" seed="20261110"}
| **Tipping point analysis**
|
| The missing values of the active groups are imputed under MAR, then shifted by delta; the smallest delta that makes
| the primary test not significant is reported.
| :::
OPEN [grounded] sensitivity SENS1 cites no protocol sentence and is no declared addition
how: cite the protocol on the line after the block: []{.cite source="protocol:<section>" words="words of the sentence"} (sapling find, or the protocol search of the editor); if the protocol does not say it, write a ::: {.decision} that leaves it open for the statistician
OPEN [agreed] sensitivity SENS1: not judged yet
how: the statistician reads it (sapling show SENS1), then: sapling agree SENS1 --by NAME, or sapling revise SENS1 "..." --by NAME

The block plans 100 imputations, a fixed seed, and delta from 0 to 6 in steps of 0.5. But Sapling scores it not grounded: it cites no protocol sentence, and nobody has declared it as an addition. The agent cannot cite a sentence that is not there (chapter 7). It must not pretend that the protocol plans it.

The Sapling assistant in the editor: an inbox of proposals on the left, and on the right a new block, a tipping-point sensitivity analysis with its estimand, method, assumption MNAR, shifts and imputations, with the buttons Accept, Edit in a form and Decline.
The same proposal in the editor: the agent's block waits in an inbox, with Accept and Decline. It compiles in its place, and it shows that no protocol sentence is cited. The study here is a made-up hypertension study.Source: the Sapling manual (sapling-sap.pages.dev)

What the SAP adds to the protocol is your decision, with your reason. You write it in the section Changes from the protocol, as a change block that names the block it adds:

sap.qmd: added at the end
# Changes from the protocol
::: {.change adds="SENS1"}
**The plan adds a tipping point analysis of the primary estimand EST1 under MNAR.**
The protocol's LOCF and its observed-case analyses make similar assumptions about the patients who stop. The tipping
point analysis shows how far the missing values of the active groups must be worse than under MAR before the
conclusion changes (ICH E9(R1) A.5.2; EMA 2010, section 7).
:::
[]{.cite source="protocol:4.3.2" words="Additionally, each of the efficacy variables will be analyzed"}
Terminal
$ sapling check
pass 1 syntax 0 error(s) 0 warning(s) PASS
pass 2 structure 0 error(s) 1 warning(s) WARN
sap.qmd:618:22: warning[software]: SAS has no version.
618 | ::: {.software name="SAS"}
| ^^^
fix: write `::: {.software name="SAS" version="..."}`: the results depend on it
pass 3 data 0 error(s) 0 warning(s) PASS
pass 4 knowledge 0 error(s) 0 warning(s) PASS
pass 5 chain 0 error(s) 0 warning(s) PASS
0 error(s), 1 warning(s)
SAPLING CHECK PASS
Terminal
$ sapling round --note "the tipping point analysis: an addition of the SAP"
...
AGENT SCORE 95/100 (+3 since round 3)
Sound 91 #########. 0 error(s), 1 warning(s) of the compiler
Grounded 100 ########## 22 of 22 deciding block(s) cite the protocol or are declared additions
Estimands 100 ########## 8 of 8 estimand check(s) hold
Rigour 100 ########## 0 error(s), 0 warning(s) of the knowledge base's rules open
Complete 80 ########.. 45 of 56 checklist item(s) shown in the plan
...
AGAINST ROUND 3: IMPROVED (estimands 88->100, complete 76->80); 3 item(s) answered
...
The whole output (24 lines)
Terminal
$ sapling round --note "the tipping point analysis: an addition of the SAP"
sap.qmd (plan 53cdf1274a8e)
AGENT SCORE 95/100 (+3 since round 3)
Sound 91 #########. 0 error(s), 1 warning(s) of the compiler
Grounded 100 ########## 22 of 22 deciding block(s) cite the protocol or are declared additions
Estimands 100 ########## 8 of 8 estimand check(s) hold
Rigour 100 ########## 0 error(s), 0 warning(s) of the knowledge base's rules open
Complete 80 ########.. 45 of 56 checklist item(s) shown in the plan
Feedback 100 ########## no note of the statistician
Executable - not run yet (sapling run): not scored
READINESS NOT READY (the statistician's)
Decisions 22 open decision(s)
Agreed 0 of 84 block(s) agreed
Approved not approved
AGAINST ROUND 3: IMPROVED (estimands 88->100, complete 76->80); 3 item(s) answered
NEXT (agent): Gamble 5: 5. Roles and responsibilities (no setting `author:` (roles and responsibilities))
how: write the block that states it (sapling generate ..., or a block of docs/markdown.md), cited; when only the statistician can say it: a ::: {.decision}
NEXT (statistician): open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47)
SAPLING EVAL: agent score 95/100; NOT READY (22 open decision(s) for the statistician; 84 of 84 block(s) not agreed); saved as round 4

Grounded is 22 of 22 again, Estimands is 100, and the agent score is 95.

Agent score measured by the compiler and the rules

95/100

  • Soundweight 20910 error(s), 1 warning(s) of the compiler
  • Groundedweight 1510022 of 22 deciding block(s) cite the protocol or are declared additions
  • Estimandsweight 201008 of 8 estimand check(s) hold
  • Rigourweight 151000 error(s), 0 warning(s) of the knowledge base's rules open
  • Completeweight 158045 of 56 checklist item(s) shown in the plan
  • Feedbackweight 10100no note of the statistician
  • Executableweight 5–not run yet (sapling run): not scored

Readiness yours: no agent can move it

NOT READY

Decisions
22 open decision(s)
Agreed
0 of 84 block(s) agreed
Approved
not approved

Next for the agent Gamble 5: 5. Roles and responsibilities (no setting `author:` (roles and responsibilities))

The scorecard after round 4, from sapling eval --json. Every agent part is at 100 except Sound (the version of the software) and Complete. Readiness has not moved: it is yours.

Your turn 8.1 Grounded again

Run sapling show SENS1 after the change block. What does the grounded item say now? What is still open?

Show the answer
Terminal
$ sapling show SENS1
sensitivity SENS1 sap.qmd:468 not judged
| ::: {.sensitivity #SENS1 estimand="EST1" method="tipping-point" assumption="MNAR" delta="0 to 6 by 0.5"
| imputations="100" seed="20261110"}
| **Tipping point analysis**
|
| The missing values of the active groups are imputed under MAR, then shifted by delta; the smallest delta that makes
| the primary test not significant is reported.
| :::
ok [grounded] sensitivity SENS1 rests on a declared addition of the SAP
OPEN [agreed] sensitivity SENS1: not judged yet
how: the statistician reads it (sapling show SENS1), then: sapling agree SENS1 --by NAME, or sapling revise SENS1 "..." --by NAME

The block now rests on a declared addition of the SAP. One item is open: you have not judged the block. Chapter 9 judges it, and asks for a change first.

What you learned

  • A sensitivity analysis tests the assumption of the primary analysis on the same estimand (ICH E9(R1) A.5.2).
  • The tipping point is the smallest shift of the missing values that changes the conclusion; you judge it clinically.
  • What the protocol does not have is an addition of the SAP: the agent proposes it, you declare it with adds= and a reason.

Next: 9 · Your judgment, block by block →

Found an error or have a question? Write to datastep.studio@gmail.com· The plan is written in Sapling; every R program runs in DataStep Studio for R.

Data: the CDISC pilot study (cdiscpilot01). The protocol as CDISC USDM from the CDISC 360i project (MIT licence); the SDTM data from the PhUSE scripts repository (MIT licence). Public example data: no real patient.

R is a project of the R Foundation. SAS is a registered trademark of SAS Institute Inc. CDISC, SDTM, ADaM and USDM are trademarks of the Clinical Data Interchange Standards Consortium. Claude is a trademark of Anthropic. This book is not affiliated with or endorsed by any of them.

© 2026 DataStep Labs. All rights reserved.