The goal of PhenoAssistant is to generate and execute suitable workflows by chaining AI models and other computational tools to address plant phenotype analysis requests from users. To achieve this, PhenoAssistant is built around a centralised multi-agent architecture, comprising an augmented LLM (the manager) and a specialised toolkit consisting of dedicated computational modules and LLM agents designed to enhance its capabilities for image-based plant phenotyping, as illustrated in Fig. 1.Fig. 1

Users provide data and task description to PhenoAssistant. The manager creates a step-by-step plan selects and executes appropriate tools, and then summarises the tool outputs to fulfil the task. Users retain full control to refine intermediate steps as needed. Manager icon made by Surang from Flaticon (www.flaticon.com). Outputs and Toolkit icons made by Freepik from Flaticon (www.flaticon.com).

In this architecture, a single manager agent is responsible for coordinating workflow creation and execution. The role of each LLM agent in the system is pre-defined and fixed. This architecture is chosen because it best supports our aim of reliable workflow generation and execution. It offers several advantages for plant phenotype analysis: it is less prone to coordination errors, produces more consistent workflows for similar task descriptions, and operates within a clearly defined set of available tools and agents. This architecture also increases transparency and user control by allowing users to intervene at any step of the generated workflow to correct errors or unexpected outcomes simply by interacting with the manager.

Specifically, upon receiving user-provided task descriptions and accompanying data, the manager first creates a plan detailing the necessary steps to fulfil the task. It then selects and executes the suitable tools with appropriate parameters, producing various types of outputs that can be saved for further analysis (either by the user or PhenoAssistant). Finally, the manager summarises these outputs, presenting essential information for task completion. Users retain control throughout this process, and can provide textual feedback to refine plans or select alternative tools and parameters, minimising errors from PhenoAssistant.

A core component of many plant phenotype analysis workflows is accurate visual information extraction. While recent generalist LLMs and agents often include vision capabilities^6,36,37, they are not always able to generalise to extracting plant phenotypes due to the limited representation of plant-specific data and tasks during their development. To demonstrate this limitation, we prompt the state-of-the-art generalist agent, ChatGPT 4o, in various ways, to extract the projected leaf area (PLA) and leaf count from an image sourced from the same dataset that will later be used in case study 1. The actual PLA and leaf count for this image are 14,260 pixels and 14, respectively.

As illustrated in Fig. 2a, ChatGPT is directly prompted to extract the requested phenotypes without additional instructions. However, it cannot correctly predict either the leaf count or PLA, despite it invokes the code writing function, attempting to use image processing techniques to solve the tasks. Furthermore, ChatGPT produces inconsistent results when processing the same prompt repeatedly (Fig. 2b). Next, as shown in Fig. 2c, we prompt ChatGPT to perform a computer vision pre-task (leaf instance segmentation), which allows for manual extraction of PLA and leaf count from the segmentation results. However, the results are inadequate for further analysis: several edges are incorrectly segmented, and not all individual leaves are successfully identified. Finally, we prompt ChatGPT to rely solely on its vision capabilities to extract the phenotypes, but the results are still unsatisfactory: tasks, such as leaf instance segmentation, cannot be performed using ChatGPT’s vision abilities (Fig. 2d), and leaf count predictions are incorrect (Fig. 2e).Fig. 2

a, b ChatGPT is prompted to extract projected leaf area and leaf count in two separate attempts, but the results are incorrect and inconsistent between attempts. c ChatGPT is prompted to perform leaf instance segmentation. It gradually refines its segmentation approach from producing no meaningful output, to generating a rough plant mask, and finally several leaf masks. However, the final output still fails to separate each leaf individually with clear boundaries. ChatGPT is prompted to rely solely on its inherent vision capabilities to perform leaf instance segmentation (d) and leaf counting (e). However, these capabilities do not support instance segmentation and also produce an incorrect leaf count. The same A. thaliana image from (a) is used as visual input for all examples. Screen captures of user-ChatGPT conversations corresponding to these examples are provided in Supplementary Fig. 1.

To overcome this limitation in generalist LLMs and agents, PhenoAssistant is equipped with phenotype extraction tools, incorporating a model zoo of computer vision models trained on plant-specific datasets, with utilities for computing traits, such as plant area and diameter, from these models’ outputs. The toolkit also supports automatic model training on new data, enabling users to expand PhenoAssistant’s vision capabilities to their unique data and analytical requirements.

Complementing these phenotype extraction tools, PhenoAssistant is also augmented with tools for phenotype analysis. This includes dedicated modules with pre-defined logic and deterministic processes for statistical tests (e.g. ANOVA). In addition, the toolkit contains various LLM agents engineered for different purposes, including the table analyser for extracting information from CSV files, the data visualiser for generating plots, and the plot analyser for interpreting and analysing generated plots. Additionally, the code writer agent supports essential tasks necessary for developing comprehensive end-to-end analysis workflows, such as merging and saving files, as well as executing novel functions not predefined within the toolkit. To enhance PhenoAssistant’s expertise in plant-related domains, a retrieval augmented generation (RAG) agent provides access to scientific literature. The Pipeline Reproducer agent enables users to extract and re-execute previously conducted phenotype extraction and analyses, including all tool calls and generated code, ensuring reproducibility for similar data.

Traditionally, executing plant phenotyping tasks requires substantial expertise in computer science, including image processing, machine learning, and coding. With PhenoAssistant, users can complete these tasks effortlessly by interacting through natural language. To demonstrate PhenoAssistant’s effectiveness, we present three case studies showcasing its ability to streamline plant phenotyping analysis.

In case study 1, we use PhenoAssistant to replicate part of the Phenotiki analysis workflow³⁸, aiming to visualise and analyse the growth patterns of different Arabidopsis thaliana ecotypes. We provide PhenoAssistant with the same dataset as in Minervini et al.³⁸, which contains 24 plants from 5 different ecotypes: wild type (Col-0, 5 plants), constitutive triple response 1 (ctr1, 5 plants), ethylene insensitive 2 (ein2.1, 5 plants), pgm (mutation in the plastidic isoform of phosphoglucomutase, 5 plants), and adh1 (mutation causing defects in alcohol dehydrogenase, 4 plants). Each plant was grown for 26 days with images captured every 12 h, resulting in a total of 1248 images.

As shown in Fig. 3, we prompt PhenoAssistant to address five relevant tasks, each including a task description, tools used by PhenoAssistant, and the generated results. We begin with the fundamental step of any plant phenotyping task, extracting phenotypes from data: in task 1, PhenoAssistant is prompted to extract phenotypes including projected leaf area (PLA) and leaf count from images. It identifies the need for a leaf instance segmentation model and hence selects the one suitable for Arabidopsis from the model zoo, executes it, and computes the requested phenotypes from the segmentation results. The code writer is then employed to write and execute Python code to merge the computed phenotypes with metadata and save them into a CSV file. To illustrate how users can interact with PhenoAssistant in practice, we provide the chat logs for case study 1 task 1 in Supplementary Note 1. At the end of task 1, the user can request PhenoAssistant to save the executed pipeline (i.e. from instance segmentation to saving the extracted phenotypes, as demonstrated at Supplementary Note 2), which can later be reapplied to a different dataset.Fig. 3

PhenoAssistant automatically completes five tasks: computing phenotypes from images, plotting phenotypic statistics, analysing a generated plot, performing statistical tests for different ecotypes, and comparing findings with literature. Each task is presented as task description (grey), tools used by PhenoAssistant (blue), and results (white). Enlarged versions of the saved plots in task 2 are provided in Supplementary Figs. 2–5.

From tasks 2 to 5, we demonstrate PhenoAssistant’s capabilities for processing and analysing the extracted phenotypes (i.e. from task 1), which are essential for identifying plant growth trends and key differences among ecotypes. In task 2, PhenoAssistant leverages the table analyser and data visualiser agents to compute and plot statistics based on user-specified plot requirements to visualise the growth patterns of different ecotypes. It can further use the plot analyser to interpret data visualisations, such as ranking ecotypes according to their PLA (Task 3). PhenoAssistant can also invoke relevant tools to conduct statistical analyses (e.g. ANOVA and Tukey–Kramer) using user-defined parameters and then summarise the key findings (task 4). For example, it categorises ecotypes by PLA size as large (ein2.1, Col-0), medium (adh1, pgm), and small (ctr1), based on Tukey–Kramer post hoc tests (p < 0.05). Furthermore, PhenoAssistant can validate these findings against previous literature by leveraging the RAG agent to embed paper contents and retrieve related knowledge (task 5). In tasks 4 and 5, PhenoAssistant performs the same statistical tests and reaches conclusions consistent with the Phenotiki paper³⁸.

In case study 2, we show that PhenoAssistant can handle other data (e.g. non-model plant) and tasks when provided with appropriate tools, such as vision models developed on other datasets. To illustrate this flexibility, we prompt PhenoAssistant to replicate part of the workflow in Williams et al.³⁹ for potato leaves, assessing the correlation between manually measured leaf area (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A$$\end{document}A) and dry weight (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W$$\end{document}W), and evaluating if this correlation holds with algorithmically computed PLA. This analysis can help plant scientists identify potential discrepancies between manual and automated measurements. For this demonstration, we integrate the leaf-only SAM model (for potato leaf segmentation) and use a dataset consisting of 32 potato images with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A$$\end{document}A and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W$$\end{document}W annotated; both the model and dataset originate from Williams et al.³⁹.

As shown in Fig. 4, PhenoAssistant is asked to compute PLA, and it follows the similar procedure to the previous case study while using a different segmentation model (task 1). It is then tasked to measure and analyse the Pearson correlation coefficient between \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A$$\end{document}A and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W$$\end{document}W, as well as PLA and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W$$\end{document}W (task 2). As this correlation analysis is not predefined in the toolkit, PhenoAssistant leverages the code writer to accomplish it. Finally, PhenoAssistant summarises the results, highlighting the correlation coefficients and providing further explanations and context. The findings indicate that while the PLA derived from leaf-only SAM is useful, it may introduce errors compared to manually measured leaf area when predicting dry weight. In task 2, PhenoAssistant performs the same correlation analysis and reaches conclusions consistent with Williams et al.³⁹.Fig. 4

In response to the user’s requests, PhenoAssistant first extracts phenotypes from the provided data and then compares correlations between different plant-related variables.

In case study 3, we demonstrate that PhenoAssistant can expand its vision capabilities via the built-in automatic vision model training function, enabling it to tackle previously unseen phenotyping challenges (e.g. experiments with new data or in new environments).

As shown in Fig. 5, PhenoAssistant is prompted to identify nutrient deficiency for winter wheats in field, where no capable model is integrated in the toolkit. Recognising the need for image classification, it instructs the user to upload labelled data in the required format. After receiving the dataset, it invokes tools to automatically preprocess the data (e.g. splitting into training and evaluation subsets, applying data augmentation), initiate model training using the user-specified finetuning method (full-parameter finetuning or low-rank adaptation (LoRA)⁴⁰, whose differences are described in the automatic vision model training subsection of the 'Methods' section), and evaluate the model’s performance. Finally, the evaluation results are presented to the user, and the trained model is added to the model zoo for future use. The data used for this case study are from Yi et al.^41,42.Fig. 5

When no suitable model is available to solve a given task, PhenoAssistant first prompts the user to provide a dataset in the desired format. The user can select between full-parameter finetuning or LoRA⁴⁰, depending on computational resource constraints and performance requirements. It then automatically applies data preprocessing, followed by training and evaluating the model. The trained model is saved in the model zoo for future use.

In summary, we present three case studies illustrating how PhenoAssistant streamlines plant phenotyping with natural language instructions, allowing plant researchers to concentrate on scientific discovery rather than technical training. These cases span various plant species, environmental conditions, and tasks, including phenotype extraction, pipeline reproduction, statistics computation and visualisation, plot analysis, statistical testing, knowledge retrieval, correlation assessment, and model training. Chat logs for all case studies are available in our code repository.

Beyond the presented case studies, we evaluate PhenoAssistant’s adaptability and potential for broader applications via a series of tasks. Since PhenoAssistant leverages rapidly evolving AI technologies, this evaluation also offers a method for assessing new AI components (e.g. DeepSeek^43,44) as replacements for PhenoAssistant’s original modules. The evaluation covers three aspects: tool selection, vision model selection, and data analysis. Prompts and detailed results for the evaluation are presented at Supplementary Notes 3–5. Chat logs for the evaluation are available in our code repository. During the evaluation, no human feedback (e.g. corrections to tool selection or tool arguments) is provided.

We evaluate the tool selection ability of PhenoAssistant by assessing whether it can select appropriate tools with the correct parameters and in the correct order to complete a task. This evaluates PhenoAssistant’s abilities of chaining multiple tools for task completion rather than the correctness of task outputs.

To conduct this evaluation, we create 20 tasks to represent various phenotyping requests with different plant species. For each task, we prompt PhenoAssistant to generate a sequence of tools with their arguments in a standalised format. The outputs from PhenoAssistant are evaluated by an LLM-based evaluator, termed tool evaluator (LLM-based evaluators have been applied in different domains^15,45). The tool evaluator is an LLM (GPT-4o) provided with detailed information about PhenoAssistant’s toolkit (e.g. descriptions of tool functionalities and arguments) and the user-specified task description. It is prompted to score the response of PhenoAssistant for each task on the following four aspects using a scale of 1–5 (where 1 is the lowest and 5 the highest):

Overall chain: whether the tool chain is reasonable for addressing the task and whether any crucial steps are missing or incorrectly ordered.

The average scores across the 20 tasks are reported in Table 1, under the row manager. Overall, PhenoAssistant consistently achieves a full score in tool existence correctness (5), indicating that it does not hallucinate tools outside its toolkit. Slightly lower scores in overall chain correctness (4.25) and argument correctness (4.3) suggest that PhenoAssistant might generate toolchains in an inappropriate order, omit certain steps, or provide less accurate arguments. These issues may also be caused by vague task descriptions, highlighting the importance of allowing users to clarify tasks through interaction with PhenoAssistant. PhenoAssistant also performs well on tool appropriateness (4.65), although this score still indicates potential limitations in the tool understanding abilities of LLMs, as well as in the clarity and comprehensiveness of the tool descriptions provided within the toolkit in PhenoAssistant.Table 1

Tool selection evaluation results under two settings (manager and manager + critic)

Beyond evaluating the original design of PhenoAssistant, we also assess the potential of introducing a critic agent. This agent is implemented as a prompted LLM (GPT-4o), provided with both the user-specified task and PhenoAssistant’s toolkit information, to critique and refine the toolchains generated by the manager. Each toolchain produced by the manager is reviewed by the critic and will be refined if issues are identified. The toolchains generated by the manager–critic interactions are assessed with the same tool evaluator in the manager-only setting.

The average scores with the critic are presented in Table 1 under the row manager + critic. The results show consistent improvements across all aspects compared to using the manager solely. This suggests that strategy verification mechanisms (e.g. adding a critic agent) can enhance the reliability of PhenoAssistant on broader tasks and reduce the burden and frequency of human verification, which should be considered in future extensions.

The system prompts for the tool evaluator and the critic are provided in Supplementary Note 6. Detailed per-task scores for the tool selection evaluation are reported in Supplementary Table 2.

Next, we evaluate whether PhenoAssistant can recommend the appropriate type of computer vision model (instance segmentation, image classification, or image regression) for a given plant phenotyping task. We term this evaluation vision model selection I. It is crucial for automatic model training, as users may not have substantial machine learning knowledge to associate plant tasks with computer vision models. Accurate suggestions from PhenoAssistant prevent users from preparing incorrect training data.

To conduct this evaluation, we create 50 plant tasks expressed in varied formats: short phrases (e.g. leaf counting), statements (e.g. highlight the weeds in these field images), and questions (e.g. what is the maturity status of these pods?). This variation aims to cover different ways users might naturally express tasks. PhenoAssistant is then asked to select the most appropriate model type corresponding to each task description. The outputs of PhenoAssistant are manually inspected.

PhenoAssistant achieves 100% accuracy in vision model selection I, indicating its potential to activate the correct model training function for a wide range of plant tasks.

In addition to evaluating PhenoAssistant’s ability to select the correct model type, we further assess whether it can identify the most suitable specific model from a vision model zoo for a given plant phenotyping task. We term this evaluation vision model selection II. It is designed to test whether PhenoAssistant can handle broader phenotyping scenarios when its model zoo expands.

To perform this evaluation, we adapt tasks from vision model selection I and create 20 tasks covering a wide range of plant species and phenotyping needs (e.g. identify all the wheat spikes in these images, and how badly are these sunflowers affected by disease? Provide a score for each image). We also create a vision model zoo containing 30 model identifiers. Each identifier follows the same naming convention used in the vision model zoo of PhenoAssistant, namely {plant-species}_{task}_{model} (e.g. wheat_spike-instance-segmentation_m2fb, sunflower_disease-severity-regress_dino2b), as described in the vision model zoo subsection of the 'Methods' section. Among these 30 model identifiers, 15 identifiers correspond exactly to 15 of the 20 plant tasks, while the remaining 15 identifiers serve as distractors.

PhenoAssistant is then asked to match each plant task with the most suitable model identifier from the model zoo. It achieves 100% accuracy in this evaluation: for the 15 tasks with corresponding model identifiers, it provides the correct match; for the remaining 5 tasks without a direct match, it not only recognises that no suitable model is available, but also recommends an appropriate model type for fine-tuning. This result demonstrates PhenoAssistant’s potential to address a wider range of plant phenotyping tasks with an extended vision model zoo.

Finally, we assess whether PhenoAssistant can produce the correct outputs for different data analysis requests. This aims to assess the reliability of the built-in LLM agents (e.g. data visualiser) for data analysis.

To conduct this evaluation, we create 20 phenotype analysis tasks using actual data. These tasks cover a range of operations including extracting values from files, computing statistics, generating plots, analysing plots, converting file formats, computing and analysing new metrics. The outputs of PhenoAssistant are manually inspected.

PhenoAssistant achieves 85% (17/20) accuracy in this evaluation. All failed cases fall within the plot analysis category. These tasks require the use of the plot analyser agent (a prompted GPT-4o) to interpret and extract fine-grained information from figures. This suggests that fine-grained visual reasoning remains a bottleneck for LLMs.

In summary, our evaluation shows that PhenoAssistant, as a multi-agent AI system, exhibits promising adaptability to broader plant phenotyping scenarios and tasks, while strategy verification mechanisms and improved visual interpretation capabilities should be further explored.

LLMs are deep neural networks with billions of parameters trained on large-scale text corpora to predict the next token in a sequence. Although this objective is simple, it enables LLMs to acquire emergent abilities, such as natural language understanding, reasoning and code generation. Recent models such as GPT-4o⁶, have also extended to multi-modal training (e.g. using both text and image data), and hence expanded their capabilities to vision-language interpretation, making them suitable to support more complex tasks.

The versatile abilities of LLMs make them effective foundations for AI agents (and hence also termed LLM agents). These agents leverage LLMs as core intelligence for reasoning, decision-making, and performing actions to achieve specified goals, such as operating structural data or retrieving knowledge from literature. This is typically achieved by providing an LLM with (i) a carefully crafted system prompt, which defines the agent’s role and acting logics (e.g. to follow specific instructions or avoid common mistakes), and (ii) access to external tools, such as computational functions, APIs, or databases, which extend the abilities of an LLM beyond those acquired from model training. When multiple agents and tools with specialised functionalities are integrated into a multi-agent AI system, the system as a whole can collectively address complex tasks that exceed the capacity of a single agent. To ensure that a such system aligns with its objectives, both system-level design (e.g. incorporating human-in-the-loop mechanisms) and element-level design (i.e. selecting appropriate agents and tools) need to be carefully considered.

PhenoAssistant comprises a manager agent and a specialised toolkit designed for plant phenotyping tasks. The manager provides an interactive interface for handling user requests, planning tasks, selecting and executing tools, and summarising results. The manager is implemented using GPT-4o (version: 2024-08-06, via OpenAI Azure API), which was the state-of-the-art LLM when PhenoAssistant was developed, with the model temperature set to 0.1, guided by a system prompt (Supplementary Note 6) that instructs it to ensure desired functionality and behaviour.

Specifically, the system prompt instructs the manager agent to (i) generate a step-by-step plan and refine it iteratively, (ii) coordinate the use of the toolkit by chaining outputs and inputs across different tools and agents, (iii) follow specialised guidelines when using specific tools or agents (e.g. the manager must verify the availability of a vision model before invoking it), and (iv) conclude interactions with a concise summary.

The toolkit consists of Python-based functions, each accompanied by structured schema (name, description, parameters, input/output format). This enables the manager agent to understand the tools’ capabilities and compose them properly into valid workflows. Notably, PhenoAssistant’s toolkit includes both deterministic modules (e.g. instance segmentation and statistical tests) and LLM agents (e.g. code writer and plot analyser).

PhenoAssistant is implemented in Python, with tool invocation and multi-agent orchestration supported by AutoGen. Computer vision models and model training functions are implemented using PyTorch and the Transformers library. The table analyser agent is implemented using Pandas AI. We introduce the key tools below; other tools, as well as a complete list of software dependencies is available in our code repository.

Many plant phenotypes cannot be extracted without the aid of image processing or computer vision techniques. Although recent LLMs have incorporated vision capabilities, they are not yet powerful enough for effective extraction of plant traits. For example, tasks such as delineating individual leaf boundaries (known as instance segmentation in computer vision), which is a preliminary task for computing leaf area and count, remain challenging.

To overcome this limitation, PhenoAssistant integrates external computer vision models. Each vision model is uniquely identified using the naming convention: {plant-species}_{task}_{(optional) training-dataset}_{model}_{(optional) finetuning-method}. These identifiers are maintained within a structured file (e.g. model_zoo.json), which the manager agent accesses whenever a vision model is required, ensuring appropriate vision model selection for the given plant task.

For demonstration purposes, in case study 1, we integrate Mask2Former⁷¹ for segmenting individual Arabidopsis leaves. Following the similar practice in Chen et al.⁷², Mask2Former is fine-tuned on subsets A1 and A4 of the publicly available CVPPP LSC dataset⁷³. In case study 2, we integrate Leaf-only SAM³⁹ for potato leaf instance segmentation.

Given the diverse conditions encountered in plant phenotyping, such as varying illumination, indoor/outdoor environments, species diversity and diverse phenotyping needs, it is practically impossible to pre-integrate every required vision model into PhenoAssistant. Currently, there is also no universal vision model capable of solving all phenotyping tasks. Furthermore, new tasks and customised user requirements continuously emerge, highlighting the need for a convenient mechanism to expand PhenoAssistant’s vision model zoo.

To satisfy this need, PhenoAssistant incorporates an automated model training pipeline for common phenotyping tasks (e.g. image classification). When users identify the need for a new vision model or when PhenoAssistant determines that its current model zoo cannot adequately solve a provided task, PhenoAssistant prompts the user to upload a dataset formatted according to predefined specifications. PhenoAssistant calls tools to automatically split the uploaded dataset into training and validation sets, as well as initiate model training. Upon completion, the newly trained model is automatically added to the vision model zoo using the naming convention described earlier and becomes available for inference.

The core idea behind this automated training pipeline is to fine-tune pre-trained vision models using plant-specific datasets⁷⁴. These models are initially trained on large-scale general data and can be fine-tuned to specific plant analysis tasks with limited additional data, thereby minimising development and deployment efforts. Specifically, for image classification (as shown in case study 3), we employ the DINOv2-base model⁷⁵ as the pre-trained model.

PhenoAssistant supports two fine-tuning strategies: low-rank adaptation (LoRA)⁴⁰ and full fine-tuning, to accommodate users with varying levels of computational resources. LoRA updates only a small subset of parameters by inserting low-rank trainable matrices into existing model layers, substantially reducing computational and memory requirements during training. This approach enables rapid adaptation at the cost of a modest performance trade-off. In contrast, full fine-tuning updates all model parameters, potentially achieving higher accuracy while at increased computational and memory demands.

Beyond phenotype extraction, PhenoAssistant supports diverse data analysis tasks, such as statistics computations, data visualisation, and interpretative plot analysis. These analyses often involve complex and customised requirements that cannot always follow pre-defined logic. For example, users may request specific visualisation styles (e.g. plotting plant growth curves for different ecotypes using specific colour) or novel analyses prompted spontaneously during exploration.

To support these dynamic and evolving analytical needs, we repurpose GPT-4o (same as the manager agent, version 2024-08-06 via the OpenAI Azure API with temperature set to 0.1) into different roles using different system prompts (Supplementary Note 6). This configuration creates a set of distinct LLM agents, each of which can be invoked by the manager when required:

Code writer: generates and executes Python code for performing tasks that are not previously defined in the toolkit. Code execution is performed with automatic error handling and retry logic. Its system prompt explicitly instructs the agent to output complete, runnable scripts.

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.