FROG analysis - a community standard to foster reproducibility and curation of constraint-based models
Constraint-based models are used to investigate metabolism in diverse conditions; particularly genome scale metabolic models (GEMs) with thousands of reactions provide opportunities to analyse organisms specific metabolism. Community standards for consistent model reconstruction, curation and sharing are crucial to ensure reproducibility, reliability and FAIR sharing of the models. MEMOTE, a community tool was developed for standardized quality assessment of the models. Here we initiate a community effort for standardised assessment of model reproducibility, which is currently lacking. Reproducibility assessment is an essential part of curation of a model, e.g. in BioModels. Following discussions at dedicated breakout sessions at HARMONY2020 (minutes of the meeting), COMBINE2020 (minutes of the meeting) and HARMONY2021 (minutes of the meeting) we have now developed FROG analysis, an ensemble of analysis of constraint-based models. We have also developed a collection of tools that generates FROG reports in a standardised schema. We propose that the modellers share FROG reports along with their model which can be used by modeller or curator to independently assess the reproducibility of a model. FROG analysis is currently used in BioModels’ workflow for the curation of constraint-based models.
A curated model should be able to faithfully reproduce the analysis results. Flux values are commonly reported as results in manuscripts with flux balance analysis of constraint-based models. These values cannot be used to test the reproducibility as often multiple solutions exist for the same. Hence, numerically reproducible results of the analysis are required to verify the reproducibility of the models. Following the discussion in the HARMONY2020 and COMBINE2020 meetings, we recognized the following list of outputs/results of the FBA analysis that are numerical reproducible and can be used for curation.1) Objective function values
1) Objective function value for a defined set of bounds should be comparable/reproducible
- Some model may not have objective function values
- There is list of often used objective function
- A detailed discussion on the biomass objective: Feist and Palsson (2010) https://www.sciencedirect.com/science/article/abs/pii/S1369527410000512
2) Flux Variability Analysis (FVA)
- FVA span min/max of flux should be comparable (for a particular objective function value)
- only small numerical differences among softwares, depending on the bound conditions.
3) Gene Deletion Fluxes
4) Reaction deletion (more coverage of reaction network)
The above FROG references will be central to the to assess the reproducibility of the model and the curation efforts.
FROG test suite
The following tools can be used to generate a FROG report.
fbc_curation (Matthias König)
a python package for FROG analysis. Currently it includes two separate implementations of the reference files generation by:
- cobrapy based on COBRApy (Constraint-Based Reconstruction and Analysis in Python) available from https://github.com/opencobra/cobrapy/
- cameo based on Cameo (Cameo—Computer Aided Metabolic Engineering and Optimization) available from https://github.com/biosustain/cameo
Command line Tools: https://github.com/matthiaskoenig/fbc_curation
Web implementation: Under development
CBMPy model curator (Brett Oliver)
Script and web-based implementation of the FROG analysis using CBMPy.
CBMPyWEB: for testing during COMBINE: http://curator.fame-vu.surf-hosted.nl:5000/index
Documentation: link to presentation
fbc_curation_matlab (Karthik Raman)
a MATLAB/COBRA helper for FROG analysis of fbc models.
Model Curation in BioModels
BioModels is one of the largest repositories of manually curated models. Model curation involves ensuring the model is (1) encoded in syntactically valid standard format such as SBML, (2) is and reproducible and (32) semantically enriched with controlled vocabularies such GO, ChEBI, etc. A model author is expected to submit a SBML model (main file) and FROG report (additional file) to BioModels. Reproducibility of the constraint-based SBML models submitted to BioModels will be verified using FROG test suite. Curators at BioModels will independently try to reproduce the FROG report using a tool different from the used by the modeller. If the results are reproducible, the model will be added to the curated branch of BioModels. Quality of the model will be tested using MEMOTE test suite and the report will be uploaded as an additional file. Furthermore, model level semantic annotations will be added to the model following MIRIAM guidelines.
E.g. Renz2020 - GEM of Human alveolar macrophage with SARS-CoV-2 ( https://www.ebi.ac.uk/biomodels/MODEL2003020001) is curated using FROG test suite.
We are preparing a manuscript “FROG analysis - a community standard to foster reproducibility and curation of constraint-based models”
If you like to contribute to this community effort and the manuscript,
Please submit one of your constraint-based models (to BioModels*) along with a FROG report generated using any one of the above FROG tools or your own tool in the standard schema.
Contact: email@example.com for any support on submission.
* If you have already submitted your model to BioModels, kindly upload the FROG report as an additional file and request to publish the model again.
Please fill the following form below
All models submitted to BioModels will be curated using the FROG tools, added to the appropriate branch and highlighted in the manuscript.
Develop a package or web application for FROG analysis.
Please contact: firstname.lastname@example.org