For the best viewing experience, consider using a tablet, laptop or desktop.

SAT Manual


Introduction


The Segmentation Annotation Tool (SAT) is an online tool designed to simplify the process of annotating EMDB Segmentation File Format (EMDB-SFF) files. EMDB-SFF is a community-driven, open file format for segmentation and subtomogram average data exchange. You can read more about EMDB-SFF at its main documentation page.

In order to use SAT you will need to have an existing segmentation in one of the supported application-specific segmentation file formats or an EMDB-SFF file.

Supported application-specific segmentation file formats supported are:

  • Amira Mesh (*.am, *.elm, *.lmb)
  • CCP4 Masks (*.map, *.mrc, *.rec)
  • IMOD (*.mod)
  • Segger (*.seg)
  • Stereolithography (STL) (*.stl)
  • SuRVoS (*.h5) (**preliminary support)
  • Amira HyperSurface (*.surf)

EMDB-SFF files contain two types of data:

  • segmentation geometries specified using either mesh, shape or volume primitives, and
  • annotations (or simply notes), which are references to accessions pertaining to publicly accessible ontological terms and related datasets.

SAT is concerned with annotations i.e. addition of the annotations to EMDB-SFF files. Annotation can also be done using sfftk (see sfftk: Alternative to SAT for how to do this).

Back to top

Quickstart


More details about each step may be found in the Using the SAT section.

Upload and Run Startup Jobs

  1. Login to SAT. If you do not have an account please begin by registering.
  2. Upload your supported segmentation by clicking the ‘New Segmentation’ link on the menu bar.
    1. Provide a name to identify your segmentation as well as select the file type from the drop down then add an optional description.
    2. Upload one or more segmentation files. The number of uploaded files depends on the file format. The following files formats allow multiple files: *.map, *.mrc, *.rec, *.stl. All other file formats require only one file.
    3. Choose at least one job from the list. We recommend that you have at least XML/HDF5 (any one format) and JSON. Selecting 3D will a run a job to ensure a 3D view of your segmentation.
  3. Submit the segmentation for processing. You can opt to have an email sent to your registered email address once the processing is complete.

Once submitted your segmentation will be listed in your list of segmentations. You will need to repeatedly refresh the page to get the status of the submitted jobs.

Jobs success

If a type of output was excluded, then no information will be provided. If all required output types show up as OK, then the corresponding output will be listed in the EMDB-SFF Segmentations section. You can click on each job to see how it was run or any errors that may have arisen.

Click here to find out more about uploading your segmentation.

Annotate

To annotate your segmentation you will need to work with an EMDB-SFF JSON file. If you selected a JSON job and it successfully completed there should be an EMDB-SFF JSON in the EMDB-SFF section of the segmentation's detail page. This will have an 'Annotate' link which launches the annotation tool.

The annotation tool has three tabs:

  • use the Metadata tab to enter details about the segmentation as a whole. Also, global external references go here.
  • use the Segments tab to enter details about individual segments. Each segment also accommodates one or more external references.
  • use the Find terms tab to search various resources for external references then add these to your segmentation.

Click here to learn out more about annotating your segmentation

Back to top

Merging

Merging is the final part of creating an annotated EMDB-SFF file. It takes the JSON and XML/HDF5 and merges the annotation with geometries to produce an EMDB-SFF XML/HDF5 (specified by the user) having both annotations and geometries.

To perform a merge do the following:

  1. From your segmentation's detail page select Run job.
  2. Select the Run completion job option.
  3. Choose Merge as the job of choice.
  4. In the merge options specify the name of the output, the output format as well as the source and merge with file options for the annotation and geometries, respectively.
  5. Select whether (default) or not you want to be notified by email once the job is completed.
  6. Confirm that all is OK then hit Submit.

Click here to learn more about merging

SAT Overview


SAT has the following main pages. Click on a thumbnail to view a larger image.

My Segmentations
My Segmentations
My Segmentations

A listing of all your segmentations ordered by creation date in reverse.

Click the thumbnail to zoom

My Segmentations
New Segmentation
New Segmentation

This page provides a set of options required to create a new segmentation.

Click the thumbnail to zoom

My Segmentations
Segmentation Details
Segmentation Details

This page displays all transactions relating to each segmentation you create. It has five sections described below.

Click the thumbnail to zoom

Segmentation Metadata Section
Segmentation Metadata Section

Displays the name, date of creation, original file format and the list of files together with action you can carry out on the segmentation.

EMDB-SFF Segmentations Section
EMDB-SFF Segmentations Section

Lists the created EMDB-SFF segmentation files. Only JSON EMDB-SFF segmentations may be annotated by clicking the 'Annotate' button then merged using the 'Merge' button.

Annotated EMDB-SFF Segmentations Section
Annotated EMDB-SFF Segmentations Section

Displays a listing of merged EMDB-SFF files which have annotations present in a JSON EMDB-SFF file in the previous section.

3D Files Section
3D Files Section

Displays the 3D metadata file if 3D views are available.

Jobs Section
Jobs Section

Shows a listing of all jobs associated with this segmentation. Jobs may be marked as either COMPLETE, FAILED, PENDING or NOT RUN.

My Segmentations
Annotate: Segmentation Metadata Tab
Annotate: Segmentation Metadata Tab

View and edit segmentation metadata.

Click the thumbnail to zoom

My Segmentations
Annotate: Segments Tab
Annotate: Segments Tab

View and edit segment annotations.

Click the thumbnail to zoom

My Segmentations
Annotate: Find Terms Tab
Annotate: Find Terms Tab

Search the available resources for terms then apply them to the appropriate sections of the segmentation (either globally or to individual segments).

Click the thumbnail to zoom

Merge job
Merge job
Merge Job

Initiate a job to merge annotation with geometry to create an annotated EMDB-SFF file

Click the thumbnail to zoom

Back to top

Using the SAT


This section describes in detail the steps to be followed when using the SAT. As mentioned earlier, SAT is designed to facilitate the process of adding annotations (or notes) to EMDB-SFF files. The main use-case would have you upload the original segmentation file(s) (a non-EMDB-SFF e.g. Segger segmentation *.seg file) then process them to return an annotated EMDB-SFF file. However, you can also upload an EMDB-SFF file (XML, HDF5 or JSON) to work with, which might considerably simplify the annotation process.

Using the SAT consists of three (3) steps:

  1. Uploading the original segmentation file
  2. Annotating the output JSON file
  3. Merging the JSON file with either an XML or HDF5 to produced either an XML or HDF5 file containing both segmentation geometries and annotations.

We now describe each of these steps in turn providing file-format specific caveats to keep in mind.

Upload the Segmentation

As mentioned previously, SAT is designed to handle the complete processing of segmentations - from native (application-specific) segmentation file formats (AS-SFF) to annotated EMDB-SFF. Alternatively, in conjunction with sfftk, you may choose to upload unannotated or partly annotated EMDB-SFF files. We deal with each of these cases separately.

Uploading A Native Segmentation

Supported File Formats

SAT currently supports the following native segmentation file formats:

  • Amira Mesh (*.am)
  • CCP4 Masks (*.map, *.mrc, *.rec)
  • IMOD (*.mod)
  • Segger (*.seg)
  • Stereolithography (STL) (*.stl)
  • SuRVoS (*.h5) (**preliminary support)
  • Amira HyperSurface (*.surf)

The best supported files formats are CCP4 Masks, IMOD, Segger and Stereolithography files. As Amira file formats are proprietary with no public documentation or APIs, support is limited and is likely to be fragile. However, we welcome example files to further improve support.

Multiple File Uploads

CCP4 masks and STL files are (generally) designed to hold only one segment per file. For this reason, we provide support for uploading multiple files for these file formats each containing a single segment.

Preparing Segmentation Files

It might be necessary to pre-process segmentation files prior to upload. This can be done using sfftk but eventually this functionality will also be available through SAT. There are several ways of preparing a segmentation. You can read more about how to perform preparation here.

  • Binning, which applies to CCP4 masks is the process of assigning only one of two values to voxels, while also reducing the number of bytes per voxel. This translates to smaller EMDB-SFF files.
  • Transforming, which applies to STL files, involving applying rotation and translations of the contained mesh. This is done to ensure proper alignment between segmentation and image.
  • Mesh reduction, applies to STL files, makes use of Paraview to substantially reduce the number of polygons for the mesh. To do this you will need to download and install the free Paraview app.

Uploading An EMDB-SFF Segmentation

If one already has an EMDB-SFF file, they can upload it directly and bypass lengthy processing.

Uploading XML or HDF5

XML or HDF5 formatted EMDB-SFF files contain geometries. Therefore, they can be used (if required) to generate a 3D view to be used during annotation.

Uploading JSON

This is the fastest way to use SAT because JSON files are typically very small and can be processed with ease. Note, however, that JSON files only hold annotations - geometry is excluded but integrating annotations from JSON to either XML or HDF5 is relatively straightforward using sfftk. Find out how to merge JSON and XML/HDF5 EMDB-SFF files here.

Running Jobs

When defining a new segmentation, you will be presented with several job options. These jobs will be run regardless of the segmentation format i.e. whether one uploads a native segmentation or an EMDB-SFF segmentation. This is because we distinguish between the uploaded file (which we want to preserve) and the EMDB-SFF files to be work on. Therefore, even if you upload a JSON file, you will still need to select JSON as the output to generate another JSON to be modified during the annotation process.

The rules of thumb are as follows:

  • Select JSON for annotation
  • Select 3D to have a 3D view of your segmentation while annotating. Note, however, that this option can only be selected if JSON and XML/HDF5 (any one or both) are also selected.
  • Select XML to have a geometry-loaded EMDB-SFF to use when doing merging as well as to generate 3D output
  • Select HDF5 to similarly have a geometry-loaded EMDB-SFF for merging.

Unless one intends to merely perform format conversion (XML to HDF5 or vice versa), annotation can only be done on JSON files.

You can run additional jobs once your segmentation has been created using the Run job button in the Segmentation Metadata section.

Annotating

Annotation appends references to external resources into the EMDB-SFF file. This is done on the annotation page of the SAT.

The annotation page has three (3) tabs: the first two tabs contain metadata about the segmentation (as a whole) and individual segments, respectively. The third tab is used to find terms from several resources.

Metadata

The first tab displays details about the entire segmentation such as:

  • a name by which the segmentation may be known;
  • the official name of the software used to create the segmentation e.g. UCSF Chimera;
  • the software version;
  • an outline of how the data was processed to produce the segmentation; and
  • a (biological) description of the segmentation.

You may also add global external references, which are accessions from curated resources relevant to the segmentation as a whole. This could be terms related to the organism/specimen and/or host environment involved, the name of the distinct procedure used to generate the data (cryo-electron microscopy), a particular biological state (e.g. infected, immortalised) or any relevant that captures an important aspect of the experiment.

Segments

The second tab displays the list of segments on the left-hand side and the details of each segment on the right-hand side. Each segment may be annotated using:

  • a name;
  • a brief description;
  • an indication on the number of instances this segment occurs;
  • the colour of the segment (read-only);

Each segment may also contain a listing of external references. External references are added in the Find terms tab.

Find Terms

The third tab displays search results from various resources. Annotating involves three (3) steps:

  • Search - Enter the search term, select the resource to search against then hit Enter;

Search resource

  • Select - From the available search results, select those to be added to the segmentation; then

Select search results

  • Assign - Select a destination for the results to be assigned to. Annotations can be added globally (to the metadata), to all segments or to the currently selected segment (from the Segments tab or the segments dropdown in the Find terms tab).

Assign search results

Search results are paged (20 per page) and pages may be navigated using the top-right button set.

Navigate search results

Autosave and Saving

The annotation page automatically saves the current annotation every 10 seconds provided a network connection is present. However, we still recommend that you manually save your annotations regularly to ensure they do not lose their work.

Save buttons

Back to top

Merging

Once you have successfully annotated the segmentation you will need to merge the annotations in the JSON into an existing XML or HDF5 EMDB-SFF file. Use the Run job button in the Segmentation Metadata section and select Run completion job to avail of the Merge job.

Note!

You have to ensure that at least one JSON and one XML/HDF5 (any one) exists before proceeding with merge.

Enter the merge parameters into the Merge Job form then click Submit. The merge job will appear in the list of jobs in the segmentation detail page.

The merged segmentation will now appear in the Annotated EMDB-SFF Segmentations section and is ready for download.

Back to top

Deleting

The various objects may be deleted using the delete button provided for each. A confirmation page is provided to check whether the deletion should occur.

Delete button

Confirm delete

Back to top

Other Ways To Use SAT


While the main use-case is based upon you uploading a native segmentation, you may also use SAT in conjunction with sfftk to speed the process of annotation. We provide several use-cases below.

Annotating An Uploaded EMDB-SFF JSON File

sfftk can be used to create a JSON file which can then be uploaded for annotation directly.

If you have a Segger segmentation and sfftk installed, you can run:

which will produce the EMDB-SFF JSON file file.json. You can then upload this file (see Upload the Segmentation) and proceed to annotate it (see Annotating).

Converting Between EMDB-SFF Formats

The easiest option to convert between EMDB-SFF formats is to use sfftk. However, if you do not have sfftk installed, you can also upload one format and request the other format as an output. For example, you can upload an XML and set HDF5 as output. Remember that converting to JSON will strip the file of all geometrical data in the output.

Downloading the Annotated EMDB-SFF JSON File

Instead of performing the merge step (outlined in Merging), you can directly download the JSON EMDB-SFF file then use sfftk to do the merging locally. For example, suppose you have downloaded the JSON file file.json and have the local file file.sff. You can then write the output to file.sff using the command:

Back to top

sfftk: Alternative to SAT


SAT uses sfftk in the background and you too can run all such commands locally. We provide only a brief synopsis of what you can do, but you can read all about sfftk at https://sfftk.readthedocs.io.

Installing sfftk is straightforward from PyPI. Consider doing so in a Python virtual environment to prevent breaking your other Python libraries.

Conversion

The sff convert command converts the supported native segmentation file formats into EMDB-SFF and is used as follows:

You can specify either the output file name:

or the output format:

More options on conversions options can be viewed by typing:

Annotation

Annotation is a bit more complex because of the three kinds of tasks involved (finding terms then viewing and modifying EMDB-SFF files). All annotation actions are handled by the sff notes subcommand. You can view the full list of subcommands using

Viewing Annotations

You can view either all notes

or only for selected segments

Additionally, you may view segmentation metadata using the -H/--header option.

or

Searching for Annotation Terms from Resources

You can search for terms from either the EBI’s Ontology Lookup Service (OLS), EMDB, UniProt or PDB.

will by default search EBI’s OLS. You can change the resource to search using the -R/--resource flag which takes either ols (default), emdb, pdb, or uniprot as arguments.

Modifying Annotations

You can add, edit, delete, clear, copy and merge annotations on the command line using sfftk. Please read the https://sfftk.readthedocs.io/en/latest/annotating.html for full details. Here we provide only a synopsis of each command.

Once annotations have been made you can either save

or trash (discard) them

Back to top


  1. EMDB-SFF: https://www.ebi.ac.uk/pdbe/emdb/emschema/segmentation.html/
  2. sfftk
    1. Availability: https://pypi.org/project/sfftk/
    2. Documentation: https://sfftk.readthedocs.io/en/latest/index.html
    3. Source Code: https://github.com/emdb-empiar/sfftk

Back to top