Benchmarking LLMs for automated systems biology model replication

Objective: Model reproducibility is a relevant issue in systems biology, mainly due to incomplete or missing information in the manuscript. Despite some attempts to address the problem with guidelines, reproducibility remains a major challenge that prevents re-using and extending published models. Recently, automated LLM-based paper to model reconstruction is gaining traction to speed-up developments and fill in missing details.

However, a key question remains: to what extent can we trust LLM-based model reconstruction?

Methods: We created SBMLLM, COSBI’s agentic system for automatically reconstructing  systems biology models from scientific literature. It is composed of an agentic reflection pipeline where a translation agent generates a mathematical model that is then simulated. Simulation errors are then handled by a critique agent, that reviews and corrects the model based on simulation errors, resulting in a final model that is compared against an expert-curated gold standard. We systematically evaluate SBMLLM on SBMLLM-Bench, an open-source collection of 75 articles whose reproducibility was manually assessed by experts of the field and for which an implementation in SBML is available. The SBML model is then automatically converted to the equivalent Antimony language, that was used to compare the performance of 6 mainstream LLMs: DeepSeek, Gemini 2.5 and 3.0 pro, ChatGPT 5.0, 5.2 and 5.2-pro.

Results: Automated LLM reproducibility score was defined as the percentage of reproduced models for which agreement with published results was achieved, where agreement is determined by at least one simulated time series attaining an Absolute Average Fold Error (AAFE) < 2.

Each manuscript was evaluated three times with identical LLM models and parameters to assess LLM generation stochasticity. We found that with SBMLLM:

1) the top-performing gemini-3.0-pro generated an executable model 97% of the times (min=95%, max=100%) but reproduced accurately (AAFE<2) only 34.5% of the models (min=32%, max=37%)

2) models from the same provider improved in newer versions (OpenAI=+4%, Google=+9.3%),

3) lowering the temperature and top-p values of LLMs consistently improved performance (+1.8%) and reduced stochasticity (−0.7%) across tested LLMs, but the effect was not statistically significant (exact permutation test, p > 0.25),

4) supplementary materials contain crucial information for reproducing published results. We observed a relevant average decrease in reproducibility when supplementary materials were not accessible: from 23.3% to 8.7% across all LLMs (min=6.6% on DeepSeek, max=20% on gemini-2.5-pro).

Finally, SBMLLM-Bench can be freely used as a baseline benchmark against which future methods can be evaluated.

Conclusion: To the best of our knowledge, this is the first systematic evaluation of LLM performance in the context of automated systems biology model replication, providing guidance on LLM model choice and parameter tuning as well as an evaluation framework that can be used to rigorously evaluate future agentic AI systems in the field.

Together with WISE, SBMLLM add another technological layer towards fast & reliable development of systems biology models.

If you are planning to create or extend a systems biology model don’t start from scratch, contact us.