Skip to content

The scorecard

In this chapter

  • Read the seven parts of the agent score and what each one measures.
  • Tell the agent score from your readiness, and say why they are never mixed.
  • Read sapling next: the agent’s items with how to answer each, then yours.
  • Use sapling status: the one screen of the study.
Time
20 minutes
At the end you have
round 1 of the loop, saved: agent score 84, NOT READY, and the ranked list of what to do next.

5.1 The idea: what a program can measure, and what it cannot

Section titled “5.1 The idea: what a program can measure, and what it cannot”

A plan has two kinds of quality. Some of it a program can check: the plan compiles, every block cites the protocol, each estimand has its five attributes, no rule of a guideline is open. The rest is judgment: is this the right estimand for the clinical question, is LOCF acceptable here, is the sentence cited the best one. Only a person can judge that.

So Sapling keeps two scores, and never mixes them:

  • The agent score (0 to 100) measures what an agent can improve by writing blocks. No AI computes it: the numbers come from the compiler, the rules of the guidelines, the plan read as data, and the last run in R.
  • Readiness is your state of the plan: the open decisions, the blocks you agreed to, the approval. No agent can move it. A high agent score is not an approval.

sapling round evaluates the plan and saves the evaluation as a round of the loop. The next round is compared with it, item by item.

Agent score measured by the compiler and the rules

84/100

  • Soundweight 20910 error(s), 1 warning(s) of the compiler
  • Groundedweight 1510019 of 19 deciding block(s) cite the protocol or are declared additions
  • Estimandsweight 20756 of 8 estimand check(s) hold
  • Rigourweight 15770 error(s), 3 warning(s) of the knowledge base's rules open
  • Completeweight 157038 of 54 checklist item(s) shown in the plan
  • Feedbackweight 10100no note of the statistician
  • Executableweight 5–not run yet (sapling run): not scored

Readiness yours: no agent can move it

NOT READY

Decisions
22 open decision(s)
Agreed
0 of 80 block(s) agreed
Approved
not approved

Next for the agent EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption

Round 1 of the pilot, from sapling eval --json. The weights are Sapling's; read the parts, not only the total.
Terminal
$ sapling round --note "the first draft"
...
AGENT SCORE 84/100
Sound 91 #########. 0 error(s), 1 warning(s) of the compiler
Grounded 100 ########## 19 of 19 deciding block(s) cite the protocol or are declared additions
Estimands 75 ########.. 6 of 8 estimand check(s) hold
Rigour 77 ########.. 0 error(s), 3 warning(s) of the knowledge base's rules open
Complete 70 #######... 38 of 54 checklist item(s) shown in the plan
Feedback 100 ########## no note of the statistician
Executable - not run yet (sapling run): not scored
READINESS NOT READY (the statistician's)
Decisions 22 open decision(s)
Agreed 0 of 80 block(s) agreed
Approved not approved
...
SAPLING EVAL: agent score 84/100; NOT READY (22 open decision(s) for the statistician; 80 of 80 block(s) not agreed); saved as round 1
The whole output (22 lines)
Terminal
$ sapling round --note "the first draft"
sap.qmd (plan f8378155fe97)
AGENT SCORE 84/100
Sound 91 #########. 0 error(s), 1 warning(s) of the compiler
Grounded 100 ########## 19 of 19 deciding block(s) cite the protocol or are declared additions
Estimands 75 ########.. 6 of 8 estimand check(s) hold
Rigour 77 ########.. 0 error(s), 3 warning(s) of the knowledge base's rules open
Complete 70 #######... 38 of 54 checklist item(s) shown in the plan
Feedback 100 ########## no note of the statistician
Executable - not run yet (sapling run): not scored
READINESS NOT READY (the statistician's)
Decisions 22 open decision(s)
Agreed 0 of 80 block(s) agreed
Approved not approved
NEXT (agent): EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption
how: write ::: {.missingdata #MD_EST1 analysis="T14_3_1_CHG" assumption="MAR" handling="..."} with the justification (EMA 2010; ICH E9(R1) A.5)
NEXT (statistician): open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47)
SAPLING EVAL: agent score 84/100; NOT READY (22 open decision(s) for the statistician; 80 of 80 block(s) not agreed); saved as round 1
Part Weight It measures
Sound 20 the plan compiles: no error, few warnings
Grounded 15 each block that decides something cites the protocol, or is an addition you declared
Estimands 20 ICH E9(R1): the five attributes of each estimand, a strategy for each intercurrent event; for a primary estimand also a missing-data handling and a sensitivity analysis
Rigour 15 no open finding of the rules of the guidelines
Complete 15 the items of the Gamble 2017 and SPIRIT 2025 checklists that a plan can show in its blocks
Feedback 10 each of your notes has the agent’s answer
Executable 5 the ADaM data, the tables and the dashboard are made in R and fresh (scored only after a run)

What to look at in round 1:

  • Estimands 75: 6 of 8 checks hold. The primary estimand has no missing-data handling and no sensitivity analysis. Chapter 7 explains both.
  • Rigour 77: the three warnings of the guideline rules from chapter 4 are open.
  • Complete 70: 38 of 54 checklist items are shown in the plan.
  • Readiness NOT READY: 22 open decisions and 0 of 80 blocks agreed.

sapling next ranks the open items. For each item of the agent it says how to answer it. Then it lists yours.

Terminal
$ sapling next --n 4
For the agent (22 open; agent score 84/100):
1. [estimands] EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption
how: write ::: {.missingdata #MD_EST1 analysis="T14_3_1_CHG" assumption="MAR" handling="..."} with the justification (EMA 2010; ICH E9(R1) A.5)
2. [estimands] EST1 (primary): no sensitivity analysis of the assumptions about the missing data (ICH E9(R1) A.5.2)
how: sapling generate sensitivity (an MNAR analysis: tipping point, or jump to reference), estimand="EST1"; what the protocol does not have is the statistician's addition: propose it in a ::: {.decision}
3. [rigour] 17. Multiplicity: the protocol names 2 primary endpoints (ADAS-Cog (11); CIBIC+), and the plan has no multiplicity strategy. (line 332)
...
For the statistician (22 open decision(s), 80 block(s) to judge): ask ONE question at a time, with the default you expect
- open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47)
- open decision (section 3.1): State the allocation ratio of the 3 groups (the allocation option of the design): the protocol does not give it, and the framework of the comparison (superiority, non-inferiority with its margin, or equivalence). (line 95)
- open decision (section 3.2): State how the patients are randomized: the method, the block size, and the stratification factors. A stratification factor becomes a term of the primary model. (line 103)
- then: sapling judge (the blocks to read and agree)
The whole output (16 lines)
Terminal
$ sapling next --n 4
For the agent (22 open; agent score 84/100):
1. [estimands] EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption
how: write ::: {.missingdata #MD_EST1 analysis="T14_3_1_CHG" assumption="MAR" handling="..."} with the justification (EMA 2010; ICH E9(R1) A.5)
2. [estimands] EST1 (primary): no sensitivity analysis of the assumptions about the missing data (ICH E9(R1) A.5.2)
how: sapling generate sensitivity (an MNAR analysis: tipping point, or jump to reference), estimand="EST1"; what the protocol does not have is the statistician's addition: propose it in a ::: {.decision}
3. [rigour] 17. Multiplicity: the protocol names 2 primary endpoints (ADAS-Cog (11); CIBIC+), and the plan has no multiplicity strategy. (line 332)
how: `::: {.multiplicity #MULT procedure="hierarchy"}` with the title "Control of the type I error" in bold, then `[...]{.hypothesis #H1 analysis="T14_3_1_CHG"}` `[...]{.hypothesis #H2 analysis="<analysis>"}`, then `:::` (or procedure=co-primary, graphical, holm; procedure=none with the reason when the protocol plans no adjustment) (source: Gamble et al. 2017, Table, item 17; ICH E9 5.6; FDA Multiple Endpoints in Clinical Trials 2022 I)
4. [rigour] 28. Missing data: the primary analysis T14_3_1_CHG (Change from baseline to Week 24) does not say how it handles missing data, or under which assumption. (line 381)
how: write `::: {.missingdata #MD_T14_3_1_CHG analysis="T14_3_1_CHG" assumption="MAR" handling="..."}` with the justification "why this handling fits the estimand" in its body and `[]{.cite source="..."}` after it (source: Gamble et al. 2017, Table, item 28; NRC report on missing data 2010, Recommendation 10; EMA missing data guideline 2010 6)
For the statistician (22 open decision(s), 80 block(s) to judge): ask ONE question at a time, with the default you expect
- open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47)
- open decision (section 3.1): State the allocation ratio of the 3 groups (the allocation option of the design): the protocol does not give it, and the framework of the comparison (superiority, non-inferiority with its margin, or equivalence). (line 95)
- open decision (section 3.2): State how the patients are randomized: the method, the block size, and the stratification factors. A stratification factor becomes a term of the primary model. (line 103)
- then: sapling judge (the blocks to read and agree)

The order is fixed: a note of yours first, then a compile error, a failed run, an error of a rule, an estimand gap, a missing citation, a warning of a rule, a checklist item. Your decisions come after the agent’s list, with a hint for the agent: ask ONE question at a time, with the default you expect.

You do not need the agent’s commands. sapling help lists the few that are yours, and sapling status is the one screen of the study: the score, your readiness, the four products, and the next thing for the agent and for you.

Terminal
$ sapling status
PILOT sap.qmd
The SAP agent score 84/100 NOT READY: 22 open decision(s) for the statistician; 80 of 80 block(s) not agreed
Products SAP NOT FINAL ADaM MISSING TLFs MISSING Dashboard MISSING
Next for the agent: EST1 (primary): no missingdata block states how missing values of its analysis are handled, and under which assumption (22 open: sapling agent)
Next for you: open decision (section 1): Name the roles and the persons: who writes this SAP, the senior statistician who reviews it, and the chief investigator or sponsor who approves it (Gamble et al. 2017, items 5 and 6). (line 47)
sapling decide
the steps of each part: sapling status --all

Every other command is the agent’s toolbox: sapling help --all lists them, each with --json.

The top of the dossier page: the agent score 95, the scorecard with a bar per part, and the statistician's part: 22 open decisions, 0 of 84 blocks agreed, not approved.
The same two scores on the dossier page that you meet in chapter 9: the agent's parts with their bars, and below them yours. Here after round 4, with agent score 95 and 22 open decisions.Source: out/eval/judge.html, written by sapling judge

Your turn 5.1 One block, several items

Look at the four items of sapling next in section 5.3. Items 1 and 4 ask for the same kind of block. Which block, and how many items of the scorecard could it answer at once?

Show the answer

A missingdata block for the primary analysis T14_3_1_CHG. It answers the estimand item of EST1 (item 1), the Gamble item 28 (item 4), and the rule on single imputation, which is the fifth item of the list. Chapter 7 writes that block, and the round after it says 5 items answered:

Terminal
$ sapling round --note "the missing data of the primary analysis"
...
AGENT SCORE 90/100 (+6 since round 1)
...
AGAINST ROUND 1: IMPROVED (estimands 75->88, rigour 77->91, complete 70->74); 5 item(s) answered
...

What you learned

  • The agent score has seven parts, each measured by a program: Sound, Grounded, Estimands, Rigour, Complete, Feedback, Executable.
  • Readiness is yours: decisions, agreements, approval. No agent can move it.
  • sapling next ranks the items, each with how to answer it; sapling status is your one screen.

Next: 6 · Working with an AI agent →

Found an error or have a question? Write to datastep.studio@gmail.com· The plan is written in Sapling; every R program runs in DataStep Studio for R.

Data: the CDISC pilot study (cdiscpilot01). The protocol as CDISC USDM from the CDISC 360i project (MIT licence); the SDTM data from the PhUSE scripts repository (MIT licence). Public example data: no real patient.

R is a project of the R Foundation. SAS is a registered trademark of SAS Institute Inc. CDISC, SDTM, ADaM and USDM are trademarks of the Clinical Data Interchange Standards Consortium. Claude is a trademark of Anthropic. This book is not affiliated with or endorsed by any of them.

© 2026 DataStep Labs. All rights reserved.