Challenge FAQ

Challenge format

The annotation guideline is available here. This guideline provides information about how to read/perform annotations in IeXML.

What are the vertical bars? What's with the numbers after the categories in some of the entries with vertical bars? How do the vertical bar types work with evaluation?

The vertical bars are used to denote more than one annotation to the span of text delimited by the e tag. This can be either disjunctions (e.g., same span could be a gene or disease but we do not know which type to assign, there are not many cases in the CALBC data) or nested/overlapping annotations. In this second case, we need to specify the pieces of text linked to each one of the annotations (either tetanus or tetanus antitoxin). Spans of text are delimited by the w tag and an identifier is assigned. These identifiers are used in the e tag to denote the tokens linked to each annotation:

<e id=":::diso:1|:::chem:1,2"><w id="1">tetanus</w> <w id="2">antitoxin</w></e>

Submission preparation

Do I just create a submission file that looks like your input file?

You have the choice of using your existing annotation system and provide annotations for the semantic groups which are using or you can base your annotations on the training data. Either way, please say in the description box of the submission site which of these two possibilities you have followed for each one of the submissions. Annotations performed with your existing system (not based on our training data) will be considered for the second harmonized set.

Should it be gzipped?

Yes. The file should be gzipped.

Can I do multiple submissions?

You can submit up to three annotated sets. You can always delete one of the submissions and upload another one.

Should I submit a file for each type?

You can submit all the semantic types in one file. You have up to three submissions and you can decide which annotations you would like to include in each one.

Do you want other types (e.g. GENIA)? If so, is there any requirement for the other types?

It is possible to provide other types. We would like to stay within the UMLS semantic groups/types but you may have a system providing annotations based on your own semantic categorization of the entities. If you have your own set of categories based on GENIA then this can be discussed during the workshop to complement the existing ones from the UMLS.

Comparison tool

Is there any README about the format of the output from the comparison tool?

We have a short introduction to the output format available here. We are working to improve it. Please do not hesitate to contact us about it.