The R Consortium Submission Working Group has spent the past several years putting modernized components through FDA pilots - R, containers, Shiny, WebAssembly, JSON - demonstrating that open-source tooling can meet regulated-submission standards. Through that work, one limitation kept surfacing: benchmark data. The CDISC SDTM/ADaM Pilot 1 dataset, our de facto test bed for every new technology component, is now nearly two decades old and narrow in scope. When we turned to AI skills for specialized statistical tasks, the gap became acute - there were not enough benchmark test cases for these workflows. Pilot 7 is our response. It has two parts: build modern, varied benchmark clinical trial data for general tooling, and crowdsource benchmark test cases for pharma-skills, an open-source library of AI skills developed jointly with Bay area Biotech-Pharma Statistics Workshop (BBSW). The first skill we open-sourced is Group Sequential Design, built on gsDesign, gsDesign2, graphicalMCP with lrstat Monte Carlo verification and a JSON-driven Word report (https://github.com/RConsortium/pharma-skills). To evaluate the skill, we aim to develop test cases that cover both typical scenarios and edge cases, helping us understand its limits across different AI agents. One week since release, the community has contributed 10+ benchmark test cases. We'd love more — especially the tough ones that stretch the skill. Submitting takes about 15 minutes on GitHub and no coding required: describe a design scenario, state the expected output, write objective rubric criteria. Each case you share helps shape how AI shows up in our field, built by the statisticians who know the work best.
Eric Zhang | Ning Leng
Eikon Therapeutics | Abbvie
From R, Shiny, and Container to AI Skills: The R Consortium Submissions Working Group Call for Contributors
The R Consortium Submission Working Group has spent the past several years putting modernized components through FDA pilots - R, containers, Shiny, WebAssembly, JSON - demonstrating that open-source tooling can meet regulated-submission standards. Through that work, one limitation kept surfacing benchmark data. The CDISC SDTM/ADaM Pilot 1 dataset, our de facto test bed for every new technology component, is now nearly two decades old and narrow in scope. When we turned to AI skills for specialized statistical tasks, the gap became acute - there were not enough benchmark test cases for these workflows. Pilot 7 is our response. It has two parts build modern, varied benchmark clinical trial data for general tooling, and crowdsource benchmark test cases for pharma-skills, an open-source library of AI skills developed jointly with Bay area Biotech-Pharma Statistics Workshop (BBSW). The first skill we open-sourced is Group Sequential Design, built on gsDesign, gsDesign2, graphicalMCP with lrstat Monte Carlo verification and a JSON-driven Word report (https//github.com/RConsortium/pharma-skills). To evaluate the skill, we aim to develop test cases that cover both typical scenarios and edge cases, helping us understand its limits across different AI agents. One week since release, the community has contributed 10+ benchmark test cases. We'd love more — especially the tough ones that stretch the skill. Submitting takes about 15 minutes on GitHub and no coding required describe a design scenario, state the expected output, write objective rubric criteria. Each case you share helps shape how AI shows up in our field, built by the statisticians who know the work best.