The scorecard
In this chapter
- Read the seven parts of the agent score and what each one measures.
- Tell the agent score from your readiness, and say why they are never mixed.
- Read
sapling next: the agent’s items with how to answer each, then yours. - Use
sapling status: the one screen of the study.
- Time
- 20 minutes
- At the end you have
- round 1 of the loop, saved: agent score 84, NOT READY, and the ranked list of what to do next.
5.1 The idea: what a program can measure, and what it cannot
Section titled “5.1 The idea: what a program can measure, and what it cannot”A plan has two kinds of quality. Some of it a program can check: the plan compiles, every block cites the protocol, each estimand has its five attributes, no rule of a guideline is open. The rest is judgment: is this the right estimand for the clinical question, is LOCF acceptable here, is the sentence cited the best one. Only a person can judge that.
So Sapling keeps two scores, and never mixes them:
- The agent score (0 to 100) measures what an agent can improve by writing blocks. No AI computes it: the numbers come from the compiler, the rules of the guidelines, the plan read as data, and the last run in R.
- Readiness is your state of the plan: the open decisions, the blocks you agreed to, the approval. No agent can move it. A high agent score is not an approval.
5.2 Round 1
Section titled “5.2 Round 1”sapling round evaluates the plan and saves the evaluation as a round of the loop. The next round is compared with it,
item by item.
Agent score measured by the compiler and the rules
84/100
Readiness yours: no agent can move it
NOT READY
- Decisions
- 22 open decision(s)
- Agreed
- 0 of 80 block(s) agreed
- Approved
- not approved
Next for the agent EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption
$ sapling round --note "the first draft"... AGENT SCORE 84/100 Sound 91 #########. 0 error(s), 1 warning(s) of the compiler Grounded 100 ########## 19 of 19 deciding block(s) cite the protocol or are declared additions Estimands 75 ########.. 6 of 8 estimand check(s) hold Rigour 77 ########.. 0 error(s), 3 warning(s) of the knowledge base's rules open Complete 70 #######... 38 of 54 checklist item(s) shown in the plan Feedback 100 ########## no note of the statistician Executable - not run yet (sapling run): not scored
READINESS NOT READY (the statistician's) Decisions 22 open decision(s) Agreed 0 of 80 block(s) agreed Approved not approved...SAPLING EVAL: agent score 84/100; NOT READY (22 open decision(s) for the statistician; 80 of 80 block(s) not agreed); saved as round 1The whole output (22 lines)
$ sapling round --note "the first draft"sap.qmd (plan f8378155fe97)
AGENT SCORE 84/100 Sound 91 #########. 0 error(s), 1 warning(s) of the compiler Grounded 100 ########## 19 of 19 deciding block(s) cite the protocol or are declared additions Estimands 75 ########.. 6 of 8 estimand check(s) hold Rigour 77 ########.. 0 error(s), 3 warning(s) of the knowledge base's rules open Complete 70 #######... 38 of 54 checklist item(s) shown in the plan Feedback 100 ########## no note of the statistician Executable - not run yet (sapling run): not scored
READINESS NOT READY (the statistician's) Decisions 22 open decision(s) Agreed 0 of 80 block(s) agreed Approved not approved
NEXT (agent): EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption how: write ::: {.missingdata #MD_EST1 analysis="T14_3_1_CHG" assumption="MAR" handling="..."} with the justification (EMA 2010; ICH E9(R1) A.5) NEXT (statistician): open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47)
SAPLING EVAL: agent score 84/100; NOT READY (22 open decision(s) for the statistician; 80 of 80 block(s) not agreed); saved as round 1| Part | Weight | It measures |
|---|---|---|
| Sound | 20 | the plan compiles: no error, few warnings |
| Grounded | 15 | each block that decides something cites the protocol, or is an addition you declared |
| Estimands | 20 | ICH E9(R1): the five attributes of each estimand, a strategy for each intercurrent event; for a primary estimand also a missing-data handling and a sensitivity analysis |
| Rigour | 15 | no open finding of the rules of the guidelines |
| Complete | 15 | the items of the Gamble 2017 and SPIRIT 2025 checklists that a plan can show in its blocks |
| Feedback | 10 | each of your notes has the agent’s answer |
| Executable | 5 | the ADaM data, the tables and the dashboard are made in R and fresh (scored only after a run) |
What to look at in round 1:
- Estimands 75: 6 of 8 checks hold. The primary estimand has no missing-data handling and no sensitivity analysis. Chapter 7 explains both.
- Rigour 77: the three warnings of the guideline rules from chapter 4 are open.
- Complete 70: 38 of 54 checklist items are shown in the plan.
- Readiness NOT READY: 22 open decisions and 0 of 80 blocks agreed.
5.3 The next thing to do
Section titled “5.3 The next thing to do”sapling next ranks the open items. For each item of the agent it says how to answer it. Then it lists yours.
$ sapling next --n 4For the agent (22 open; agent score 84/100): 1. [estimands] EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption how: write ::: {.missingdata #MD_EST1 analysis="T14_3_1_CHG" assumption="MAR" handling="..."} with the justification (EMA 2010; ICH E9(R1) A.5) 2. [estimands] EST1 (primary): no sensitivity analysis of the assumptions about the missing data (ICH E9(R1) A.5.2) how: sapling generate sensitivity (an MNAR analysis: tipping point, or jump to reference), estimand="EST1"; what the protocol does not have is the statistician's addition: propose it in a ::: {.decision} 3. [rigour] 17. Multiplicity: the protocol names 2 primary endpoints (ADAS-Cog (11); CIBIC+), and the plan has no multiplicity strategy. (line 332)...For the statistician (22 open decision(s), 80 block(s) to judge): ask ONE question at a time, with the default you expect - open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47) - open decision (section 3.1): State the allocation ratio of the 3 groups (the allocation option of the design): the protocol does not give it, and the framework of the comparison (superiority, non-inferiority with its margin, or equivalence). (line 95) - open decision (section 3.2): State how the patients are randomized: the method, the block size, and the stratification factors. A stratification factor becomes a term of the primary model. (line 103) - then: sapling judge (the blocks to read and agree)The whole output (16 lines)
$ sapling next --n 4For the agent (22 open; agent score 84/100): 1. [estimands] EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption how: write ::: {.missingdata #MD_EST1 analysis="T14_3_1_CHG" assumption="MAR" handling="..."} with the justification (EMA 2010; ICH E9(R1) A.5) 2. [estimands] EST1 (primary): no sensitivity analysis of the assumptions about the missing data (ICH E9(R1) A.5.2) how: sapling generate sensitivity (an MNAR analysis: tipping point, or jump to reference), estimand="EST1"; what the protocol does not have is the statistician's addition: propose it in a ::: {.decision} 3. [rigour] 17. Multiplicity: the protocol names 2 primary endpoints (ADAS-Cog (11); CIBIC+), and the plan has no multiplicity strategy. (line 332) how: `::: {.multiplicity #MULT procedure="hierarchy"}` with the title "Control of the type I error" in bold, then `[...]{.hypothesis #H1 analysis="T14_3_1_CHG"}` `[...]{.hypothesis #H2 analysis="<analysis>"}`, then `:::` (or procedure=co-primary, graphical, holm; procedure=none with the reason when the protocol plans no adjustment) (source: Gamble et al. 2017, Table, item 17; ICH E9 5.6; FDA Multiple Endpoints in Clinical Trials 2022 I) 4. [rigour] 28. Missing data: the primary analysis T14_3_1_CHG (Change from baseline to Week 24) does not say how it handles missing data, or under which assumption. (line 381) how: write `::: {.missingdata #MD_T14_3_1_CHG analysis="T14_3_1_CHG" assumption="MAR" handling="..."}` with the justification "why this handling fits the estimand" in its body and `[]{.cite source="..."}` after it (source: Gamble et al. 2017, Table, item 28; NRC report on missing data 2010, Recommendation 10; EMA missing data guideline 2010 6)
For the statistician (22 open decision(s), 80 block(s) to judge): ask ONE question at a time, with the default you expect - open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47) - open decision (section 3.1): State the allocation ratio of the 3 groups (the allocation option of the design): the protocol does not give it, and the framework of the comparison (superiority, non-inferiority with its margin, or equivalence). (line 95) - open decision (section 3.2): State how the patients are randomized: the method, the block size, and the stratification factors. A stratification factor becomes a term of the primary model. (line 103) - then: sapling judge (the blocks to read and agree)The order is fixed: a note of yours first, then a compile error, a failed run, an error of a rule, an estimand gap, a missing citation, a warning of a rule, a checklist item. Your decisions come after the agent’s list, with a hint for the agent: ask ONE question at a time, with the default you expect.
5.4 One screen for you
Section titled “5.4 One screen for you”You do not need the agent’s commands. sapling help lists the few that are yours, and sapling status is the one
screen of the study: the score, your readiness, the four products, and the next thing for the agent and for you.
$ sapling statusPILOT sap.qmd
The SAP agent score 84/100 NOT READY: 22 open decision(s) for the statistician; 80 of 80 block(s) not agreed Products SAP NOT FINAL ADaM MISSING TLFs MISSING Dashboard MISSING
Next for the agent: EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption (22 open: sapling agent) Next for you: open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47) sapling decide
the steps of each part: sapling status --allEvery other command is the agent’s toolbox: sapling help --all lists them, each with --json.

Your turn 5.1 One block, several items
Look at the four items of sapling next in section 5.3. Items 1 and 4 ask for the same kind of block. Which block, and
how many items of the scorecard could it answer at once?
Show the answer
A missingdata block for the primary analysis T14_3_1_CHG. It answers the estimand item of EST1 (item 1), the
Gamble item 28 (item 4), and the rule on single imputation, which is the fifth item of the list. Chapter 7 writes that
block, and the round after it says 5 items answered:
$ sapling round --note "the missing data of the primary analysis"... AGENT SCORE 90/100 (+6 since round 1)... AGAINST ROUND 1: IMPROVED (estimands 75->88, rigour 77->91, complete 70->74); 5 item(s) answered...What you learned
- The agent score has seven parts, each measured by a program: Sound, Grounded, Estimands, Rigour, Complete, Feedback, Executable.
- Readiness is yours: decisions, agreements, approval. No agent can move it.
sapling nextranks the items, each with how to answer it;sapling statusis your one screen.
Next: 6 · Working with an AI agent →
Found an error or have a question? Write to datastep.studio@gmail.com· The plan is written in Sapling; every R program runs in DataStep Studio for R.
Data: the CDISC pilot study (cdiscpilot01). The protocol as CDISC USDM from the CDISC 360i project (MIT licence); the SDTM data from the PhUSE scripts repository (MIT licence). Public example data: no real patient.
R is a project of the R Foundation. SAS is a registered trademark of SAS Institute Inc. CDISC, SDTM, ADaM and USDM are trademarks of the Clinical Data Interchange Standards Consortium. Claude is a trademark of Anthropic. This book is not affiliated with or endorsed by any of them.
© 2026 DataStep Labs. All rights reserved.