We introduce {llumen}, an R package used at Merck KGaA, Darmstadt, Germany that implements an agentic framework bases on {ellmer} and enables pharmaceutical researchers to work with biomedical documents, large real-world evidence and biomarker databases, as well as with biological foundation models using large language models as orchestrators.

{llumen} currently supports the following functionalities:

**Large language models**

- Runs on standard company laptops (Windows, Linux, or macOS).
- Allows using a wide range of private (Azure), public (OpenAI, Anthropic, Bedrock), and local (llama.cpp) LLMs.
- Provides both programmatic and fully interactive (chatbot) ways to interact with the LLM.
- Supports chain-of-thoughts, tree-of-thoughts, and ReAct reasoning strategies.
- Supports context management to maintain conversation history and state.

**Text analysis**

- Supports embeddings (Azure, public, or local) using a local, high-performance vector DB (based on {ragnar}).
- Implements chunking office documents (Word, Excel, Powerpoint, RTF, PDF) into the vector DB.
- While reading office documents, supports extracting tables with summary data.
- Enables retrieval-augmented generation (RAG) using the vector DB, either explicitly or by using a RAG tool that the LLM calls.
- Emulates the Future House's PaperQA2 approach for summarizing and re-ranking parts of semantically related documents so that answer quality and comprehensiveness is significantly boosted.
- Extracts text from documents to fill structured templates, such as the JSON-based {llumen} Analysis Results Schema (LARS). Extracted JSONs are directly validated by the LLM using an R tool.
- Supports automatic diagram generation from texts, for instance to capture the main concepts of a paper in two or three flow charts.
- Provides advanced text preprocessing and cleaning capabilities for various document types.
- Offers customizable tokenization and text segmentation methods.

**Agentic tool use and codegen**

- Allows the LLM to search the internet via Google and access the textual content of websites in a controlled manner. Also includes specialized query tools for Pubmed, Pubchem, and Wikipedia.
- Enables the LLM agent to query Parquet files, Neo4j knowledge graphs, and GraphQL APIs using SQL, Cypher and GraphQL codegen in a secure context. The database agents are very capable and can answer complex, open-ended scientific questions using these data sources.
- Enables the LLM agent to use local foundation models, such as Google's TxGemma model for pharmaceutical applications, for instance for predicting solubility of small molecules.
- Uses multimodal LLMs to accurately diagnose cancer histopathology images with very comprehensive pathology reports that investigate architectural and cytological Features of H&E slides.
- Makes it easy to give the LLM new R tool capabilities (basically all R functions), share tools via MCP (built-in pure R MCP server), or utilize tools of external MCP server as internal tools (via a MCP tool wrapper).
- Supports dynamic tool creation and registration during runtime.
- Provides a framework for defining custom tools and integrating them seamlessly with the LLM.

We are internally using the package for the following usecases:

**Supported usecase**

1. Extracting results of statistical analyses from office documents: The vignettes show how to configure an agentic LLM to extract chunks oftext from documents and tables, semantically query a vector store (agent-based retrieval-augmented generation, RAG), load JSON definitions, extract JSON-structured data from documents, and validate these definitions against a user-defined JSON template. {llumen} can be utilized to extract results data from a large corpus of clinical trial results files. Templates for both CDISC Analysis Result Standard (ARS) and the {llumen} Analysis Result Schema (LARS) are provided.
2. Querying tabular databases and knowledge graphs: {llumen} excels at SQL codegen, allowing users to query very large Parquet files located on the user's device and perform complex data analyses. It also supports querying Neo4j knowledge graphs and GraphQL APIs, enabling comprehensive data exploration and analysis.
3. Working with biological foundation models: {llumen}'s ability to use local foundation models specialized for particular biological or pharmaceutical applications, such as Google's GemmaTX, enhances agentic LLMs with specialized capabilities for tasks like predicting molecular properties or analyzing biological pathways.
4. Automated literature review and summarization: {llumen} can be used to perform comprehensive literature reviews by searching multiple databases (e.g., PubMed, OpenAlex), extracting relevant information, and generating structured summaries of findings.
5. Histopathology image analysis: The package supports the use of multimodal LLMs for analyzing histopathology images, providing detailed reports on architectural and cytological features of H&E slides, which can aid in cancer diagnosis and research.
6. Drug discovery and development support: By integrating various tools and databases (e.g., PubChem, TxGemma), {llumen} can assist in various stages of drug discovery, from initial compound screening to predicting drug properties and potential interactions.
7. Clinical trial data analysis and reporting: {llumen}'s capabilities in extracting and structuring data from various document types make it well-suited for analyzing and reporting on clinical trial data, potentially accelerating the process of deriving insights from trial results.
8. Biomedical knowledge graph exploration: The package's ability to query knowledge graphs allows researchers to explore complex relationships in biomedical data, potentially uncovering new insights or research directions.
9. Automated scientific writing assistance: {llumen} can be used to assist in scientific writing tasks, such as generating literature summaries, creating structured abstracts, or even drafting sections of scientific papers based on analyzed data and literature.
10. Regulatory document preparation and analysis: The package's capabilities in extracting and structuring information from various document types can be applied to preparing and analyzing regulatory documents in the pharmaceutical industry
Sven-Eric Schelhorn
Merck KGaA
{llumen}: An agentic LLM framework for biomedical documents, databases & foundation models
We introduce  llumen , an R package used at Merck KGaA, Darmstadt, Germany that implements an agentic framework bases on  ellmer  and enables pharmaceutical researchers to work with biomedical documents, large real-world evidence and biomarker databases, as well as with biological foundation models using large language models as orchestrators.   llumen  currently supports the following functionalities  **Large language models**  - Runs on standard company laptops (Windows, Linux, or macOS). - Allows using a wide range of private (Azure), public (OpenAI, Anthropic, Bedrock), and local (llama.cpp) LLMs. - Provides both programmatic and fully interactive (chatbot) ways to interact with the LLM. - Supports chain-of-thoughts, tree-of-thoughts, and ReAct reasoning strategies. - Supports context management to maintain conversation history and state.  **Text analysis**  - Supports embeddings (Azure, public, or local) using a local, high-performance vector DB (based on  ragnar ). - Implements chunking office documents (Word, Excel, Powerpoint, RTF, PDF) into the vector DB. - While reading office documents, supports extracting tables with summary data. - Enables retrieval-augmented generation (RAG) using the vector DB, either explicitly or by using a RAG tool that the LLM calls. - Emulates the Future House's PaperQA2 approach for summarizing and re-ranking parts of semantically related documents so that answer quality and comprehensiveness is significantly boosted. - Extracts text from documents to fill structured templates, such as the JSON-based  llumen  Analysis Results Schema (LARS). Extracted JSONs are directly validated by the LLM using an R tool. - Supports automatic diagram generation from texts, for instance to capture the main concepts of a paper in two or three flow charts. - Provides advanced text preprocessing and cleaning capabilities for various document types. - Offers customizable tokenization and text segmentation methods.  **Agentic tool use and codegen**  - Allows the LLM to search the internet via Google and access the textual content of websites in a controlled manner. Also includes specialized query tools for Pubmed, Pubchem, and Wikipedia. - Enables the LLM agent to query Parquet files, Neo4j knowledge graphs, and GraphQL APIs using SQL, Cypher and GraphQL codegen in a secure context. The database agents are very capable and can answer complex, open-ended scientific questions using these data sources. - Enables the LLM agent to use local foundation models, such as Google's TxGemma model for pharmaceutical applications, for instance for predicting solubility of small molecules. - Uses multimodal LLMs to accurately diagnose cancer histopathology images with very comprehensive pathology reports that investigate architectural and cytological Features of H&E slides. - Makes it easy to give the LLM new R tool capabilities (basically all R functions), share tools via MCP (built-in pure R MCP server), or utilize tools of external MCP server as internal tools (via a MCP tool wrapper). - Supports dynamic tool creation and registration during runtime. - Provides a framework for defining custom tools and integrating them seamlessly with the LLM.  We are internally using the package for the following usecases  **Supported usecase**  1. Extracting results of statistical analyses from office documents The vignettes show how to configure an agentic LLM to extract chunks oftext from documents and tables, semantically query a vector store (agent-based retrieval-augmented generation, RAG), load JSON definitions, extract JSON-structured data from documents, and validate these definitions against a user-defined JSON template.  llumen  can be utilized to extract results data from a large corpus of clinical trial results files. Templates for both CDISC Analysis Result Standard (ARS) and the  llumen  Analysis Result Schema (LARS) are provided. 2. Querying tabular databases and knowledge graphs  llumen  excels at SQL codegen, allowing users to query very large Parquet files located on the user's device and perform complex data analyses. It also supports querying Neo4j knowledge graphs and GraphQL APIs, enabling comprehensive data exploration and analysis. 3. Working with biological foundation models  llumen 's ability to use local foundation models specialized for particular biological or pharmaceutical applications, such as Google's GemmaTX, enhances agentic LLMs with specialized capabilities for tasks like predicting molecular properties or analyzing biological pathways. 4. Automated literature review and summarization  llumen  can be used to perform comprehensive literature reviews by searching multiple databases (e.g., PubMed, OpenAlex), extracting relevant information, and generating structured summaries of findings. 5. Histopathology image analysis The package supports the use of multimodal LLMs for analyzing histopathology images, providing detailed reports on architectural and cytological features of H&E slides, which can aid in cancer diagnosis and research. 6. Drug discovery and development support By integrating various tools and databases (e.g., PubChem, TxGemma),  llumen  can assist in various stages of drug discovery, from initial compound screening to predicting drug properties and potential interactions. 7. Clinical trial data analysis and reporting  llumen 's capabilities in extracting and structuring data from various document types make it well-suited for analyzing and reporting on clinical trial data, potentially accelerating the process of deriving insights from trial results. 8. Biomedical knowledge graph exploration The package's ability to query knowledge graphs allows researchers to explore complex relationships in biomedical data, potentially uncovering new insights or research directions. 9. Automated scientific writing assistance  llumen  can be used to assist in scientific writing tasks, such as generating literature summaries, creating structured abstracts, or even drafting sections of scientific papers based on analyzed data and literature. 10. Regulatory document preparation and analysis The package's capabilities in extracting and structuring information from various document types can be applied to preparing and analyzing regulatory documents in the pharmaceutical industry