FROG analysis - a community standard to foster reproducibility and curation of constraint-based models

 

Summary

Constraint-based models are used to investigate metabolism in diverse conditions; particularly genome scale metabolic models (GEMs) with thousands of reactions provide opportunities to analyse organisms specific metabolism. Community standards for consistent model reconstruction, curation and sharing are crucial to ensure reproducibility, reliability and FAIR sharing of the models. MEMOTE, a community tool was developed for standardized quality assessment of the models. Here we initiate a community effort for standardised assessment of model reproducibility, which is currently lacking.  Reproducibility assessment is an essential part of curation of a model, e.g. in BioModels. Following discussions at dedicated breakout sessions at HARMONY2020 (minutes of the meeting), COMBINE2020 (minutes of the meeting) and HARMONY2021 (minutes of the meeting) we have now developed FROG analysis, an ensemble of analysis of constraint-based models. We have also developed a collection of tools that generates FROG reports in a standardised schema. We propose that the modellers share FROG reports along with their model which can be used by modeller or curator to independently assess the reproducibility of a model. FROG analysis is currently used in BioModels’ workflow for the curation of constraint-based models. 

 

FROG analysis

A curated model should be able to faithfully reproduce the analysis results. Flux values are commonly reported as results in manuscripts with flux balance analysis of constraint-based models. These values cannot be used to test the reproducibility as often multiple solutions exist for the same. Hence, numerically reproducible results of the analysis are required to verify the reproducibility of the models. Following the discussion in the HARMONY2020 and COMBINE2020 meetings, we recognized the following list of outputs/results of the FBA analysis that are numerical reproducible and can be used for curation.1) Objective function values 

1) Objective function value for a defined set of bounds should be comparable/reproducible

2) Flux Variability Analysis (FVA)

  • FVA span min/max of flux should be comparable (for a particular objective function value)
  • only small numerical differences among softwares, depending on the bound conditions.

‚Äč3) Gene Deletion Fluxes

4) Reaction deletion (more coverage of reaction network)

The above FROG references will be central to the to assess the reproducibility of the model and the curation efforts.

 

FROG test suite

The following tools can be used to generate a FROG report. 

  1. fbc_curation (Matthias König)

a python package for FROG analysis. Currently it includes two separate implementations of the reference files generation by: 

  • cobrapy based on COBRApy (Constraint-Based Reconstruction and Analysis in Python) available from https://github.com/opencobra/cobrapy/
  • cameo based on Cameo (Cameo—Computer Aided Metabolic Engineering and Optimization) available from https://github.com/biosustain/cameo

Command line Tools: https://github.com/matthiaskoenig/fbc_curation

Documentation: https://fbc-curation.readthedocs.io/en/latest/index.html

Web implementation:  Under development 

  1. CBMPy model curator (Brett Oliver)

Script and web-based implementation of the FROG analysis using CBMPy.

CBMPyWEB: for testing during COMBINE: http://curator.fame-vu.surf-hosted.nl:5000/index

CBMPy: https://github.com/SystemsBioinformatics/cbmpy

Documentation: link to presentation

  1. fbc_curation_matlab (Karthik Raman)
    a MATLAB/COBRA helper for FROG analysis of fbc models.

https://github.com/RamanLab/fbc_curation_matlab

 

Model Curation in BioModels

 BioModels is one of the largest repositories of manually curated models. Model curation involves ensuring the model is (1) encoded in syntactically valid standard format such as SBML, (2) is and reproducible and (32) semantically enriched with controlled vocabularies such GO, ChEBI, etc. A model author is expected to submit a SBML model (main file) and FROG report (additional file) to BioModels. Reproducibility of the constraint-based SBML models submitted to BioModels will be verified using FROG test suite. Curators at BioModels will independently try to reproduce the FROG report using a tool different from the used by the modeller. If the results are reproducible, the model will be added to the curated branch of BioModels. Quality of the model will be tested using MEMOTE test suite and the report will be uploaded as an additional  file. Furthermore, model level semantic annotations will be added to the model following MIRIAM guidelines. 

E.g. Renz2020 - GEM of Human alveolar macrophage with SARS-CoV-2 ( https://www.ebi.ac.uk/biomodels/MODEL2003020001 is curated using FROG test suite.

 

Community manuscript

We are preparing a manuscript “FROG analysis - a community standard to foster reproducibility and curation of constraint-based models” 

If you like to contribute to this community effort and the manuscript, 

  1. Please submit one of your constraint-based models (to BioModels*) along with a FROG report generated using any one of the above FROG tools or your own tool in the standard schema.

Contact: biomodels-cura@ebi.ac.uk for any support on submission.

* If you have already submitted your model to BioModels, kindly upload the  FROG report as an additional file and request to publish the model again. 

Please fill the following form below
https://docs.google.com/forms/d/e/1FAIpQLSca8t77y_bn85zEbVvKBtylN_UnHSurKBCnPTtMTpyVBnpBZA/viewform

All models submitted to BioModels will be curated using the FROG tools, added to the appropriate branch and highlighted in the manuscript.

  1. Develop a package or web application for FROG analysis.
    Please contact: sheriff@ebi.ac.uk