Get   for     ? 
 Site search     ? 
CAPRI: Critical Assessment of PRediction of Interactions
   Call For Targets
   Meeting Report
   Evaluation Meeting
   Evaluation Meeting
   ROUND 1
   ROUND 2
 MSD  CAPRI: Critical Assessment of PRediction of Interactions

First community wide experiment on the comparative evaluation of protein-protein docking for structure prediction

Hosted By EMBL/EBI-MSD Group

Scoring Discussion

Date: Tue, 11 Sep 2001 13:38:27 +0100 (BST)
From: Graham Smith
Subject: capri submission

Dear Kim,

I just wanted to check a few things about the submission format before finalizing my submissions. I see from the submissions page that a script will automatically check the complexes but I would like to check what's going on before I submit them as it might otherwise lead to a scramble to meet the deadline!

In particular

Do I have to resubmit the haemagglutinin antibody (3rd target) as it came, with atom numbers starting from 11680, and a strange atom order within residues?

The kinase in the 1st target came with some sidechains missing (not resolved), which I built on before docking. Do I have to remove them again?

The substrate in the 1st target was a S46D mutant, which I mutated back to S46 before docking. Do I have to put it back as D? (I imagine the answer to this one at least is yes)




From Fri Sep 14 08:26:40 2001
Date: Tue, 11 Sep 2001 14:21:34 +0100 (BST)
Subject: Re: capri submission

Dear Graham,

re the targets - these were all as supplied by Joel and put into chain ID order by us at the EBI

Certainly the atom serial number is of no consequence and I cannot see this being used in any way.

as far as the atom order goes all submissions are processed to a standard pdb style before we are send them on to the accessors e.g. traget 03 will what ever order given and whether it will have H's will become

ATOM  15588  N   GLN L   1 
ATOM  15589  CA  GLN L   1 
ATOM  15590  C   GLN L   1 
ATOM  15591  O   GLN L   1 
ATOM  15592  CB  GLN L   1 
ATOM  15593  CG  GLN L   1 
ATOM  15594  CD  GLN L   1 
ATOM  15595  OE1 GLN L   1 
ATOM  15597  NE2 GLN L   1 

(from the supplied target file of

ATOM  11680  CB  GLN L   1
ATOM  11681  CG  GLN L   1
ATOM  11682  CD  GLN L   1
ATOM  11683  OE1 GLN L   1
ATOM  11684  NE2 GLN L   1
ATOM  11685  C   GLN L   1
ATOM  11686  O   GLN L   1
ATOM  11687  N   GLN L   1
ATOM  11688  CA  GLN L   1  

which also has the occupancy in wrong columns)

re target 01
modelling in of complete side chains is a plus I would think the target set minus some side chains is the observed density

re the S46D mutant - as far as I understood from Joel the coordinates supplied were for what was the actual structures that have been determined and Ser46 is rather crucial to the mechanism but the ASP is the un-published structure in 1JB1 (un-complexed) - I dont have infomation if the complex is the Ser46Asp mutant or the native Ser46 The target docked complex is not specially noted to have ADP and I imagine the ASP mutant is so one can isolate the ADP in the crystal - similarly the target is supplied as the Seleno Met and the docked target is likely to molecular replacement so would be the MET rather than MSE, as HPr-Ser46 is suggested to have key importance in the interface with the kinase then the target protein may well be the Ser46 and may play an important role in the docking

These differences Ser46/Asp46 and MSE/MET however should not in my opinion be significant in the Assessment

I suggest you leave it as Ser

as far as the deadline - we will probably be open for submissions for a couple of days after the 16th


Date: Tue, 11 Sep 2001 15:04:33 +0100 (BST)
From: Graham Smith
Subject: Re: capri submission

Dear Kim,

Thanks for the message and the help. (I had forgotten about the MSE residues, which I had changed to MET). I'll put in the submissions along the lines you suggest.

Thanks again, 


From: "E.J.Gardiner"
Date: Tue, 11 Sep 2001 09:30:21 +0100
Subject: Capri submission

Dear Kim

Apologies if you have already replied to a similar e-amil - if so I haven't received it. In docking target 1, we have used A different HPR, pdb code 1ptf, rather than 1sph as supplied by the contest. Do you wish us to superpose 1sph onto our models? It will (hopefully) make no to the answer, but might different residues throw an automatic checking procedure?

Dr Eleanor Gardiner  

Date: Tue, 11 Sep 2001 09:52:42 +0100 (BST)
From: Kim Henrick 
Subject: Re: Capri submission

Dear Eleanor,

In Joel's original note he did say one could use chains from different pdb entries to those supplied in the target files - so this should be allowed. However for our automatic first stage checking procedures to get the submission into a standard form that should make subsequent accessments easier we assumed that people would start from the target xyz files on the capri web pages - so that what ever accessment was to be carried out after we at the ebi pass on the files to the accessors would be more suitable to an automatic protocol.

I havent checked but is the sequence for 1ptf the same as 1sph ? - the main thing would be to get the chain ID's to match and the model records included, then as Joel allowed this in the first announcement then I assume that the assessors have thought of this and have a procedure that will be able to cope with what you have.


Date: Thu, 13 Sep 2001 04:02:05 -0700
From: Miri Eisenstein
Subject: Re: CAPRI scoring

Hello everybody,

It is too late, in my opinion, to change the mode of evaluation and scoring for the current CAPRI experiment. So, the predictions should be evaluated with a cutoff distance of 4 Angstroms, for better for worse. However, Shoshana's suggestion to test other distance cutoffs and the sensitivity of the results to different cutoffs should be favorably considered. It will help us, the predictors, and will provide information for the future CAPRI's.


Date: Thu, 13 Sep 2001 01:02:07 -0700
From: Julie Mitchell
Subject: Re: CAPRI scoring question


Thanks for the prompt response. I personally think that in the absence of hydrogens that a 4 Angstrom cutoff for residue contacts is about 1 Angstrom too small. It effectively encourages participants to submit structures that would have collisions when modeled properly with all the hydrogens.

As a minor side note, I think the name "Critial Assessment of Predicted Interactions" reads a little better than "Critical Assessment of Prediction of Interactions"

With kind regards,

Date: Thu, 13 Sep 2001 09:56:21 +0200 (MET DST)
From: Joel Janin
Subject: Re: CAPRI scoring question

Dear Julie,

'our' criterion, as you cite it, is taken from M. Sternberg's proposal. As none of the participants has challenged it, I expect Shoshana will use it as it stands - ignoring hydrogens. I do not expect the predictions to reach the level of accuracy where it makes a difference!



On Wed, 12 Sep 2001, Julie Mitchell wrote:

Shoshana and Joel,

I have a simple question regarding the scoring of residue contacts for the CAPRI targets. As I understand your criterion, two residues are deemed in contact provided there is a pair of atoms (one from each residue) that are within 4 Angstoms distance of one another. My question is whether hydrogen atoms are taken into account or whether only hard atoms are used in determining residue contacts.

Thanks in advance,
Julie Mitchell

Date: Thu, 13 Sep 2001 01:59:21 -0700
From: Julie Mitchell
Subject: CAPRI scoring

Dear fellow CAPRI participants,

An issue has come up regarding the scoring of residue contacts for CAPRI, and Joel Janin has suggested I circulate something to all the participants.

As you are likely aware, the scoring of predictions is based on residue contacts. The question I asked of the organizing committee is whether hydrogen atoms will be used in determining residue contacts. The response I got back was that only non-hydrogen atoms will be used. My opinion is that a 4 Angstrom cutoff in the absence of hydrogens is too small, and I suggest that 5 Angstroms is a more reasonable cutoff in this case.

Shoshana has just now suggested that they intend to review how the results change as the cutoff is varied. I do, however, think that changing the scoring criteria midstream will cause arguments and that it is best to have a specific cutoff going in that people agree on.

If you have an opinion on this issue, please forward your comments to Joel Janin and Shoshana Wodak

Best wishes (and good luck with your predictions),
Julie Mitchell

PS: I am appending a copy of email correspondence with Joel and Shoshana

Joel Janin wrote:


Thanks for the prompt response. I personally think that in the absence of hydrogens that a 4 Angstrom cutoff for residue contacts is about 1 Angstrom too small. It effectively encourages participants to submit structures that would have collisions when modeled properly with all the hydrogens.

You are welcome to circulate the suggestion to other participants.

As a minor side note, I think the name "Critial Assessment of Predicted Interactions" reads a little better than "Critical Assessment of Prediction of Interactions"

I fully suscribe to that suggestion.
With kind regards,

Shoshana Wodak wrote:

Dont worry. In any case we would have tested several cutoff, just to see how results change. But you are welcome to make your proposal.

I just rechecked the website, and hydrogens aren't mentioned anywhere in the documents. I was pretty sure I'd read them carefully before. Thanks again for clarifying the details of the organizing committee's criteria.

Shoshana Wodak wrote:
Dear Julie,

Given the accuracy of most calculations, it seems reasonable not to consider hyrdogen atoms, as stated in the original proposal for the evaluation.

Date: Thu, 13 Sep 2001 18:06:34 +0200 (MET DST)
From: Joel Janin
Subject: Re: scoring

Dear Elizabeth,

the rules were up to now:
- 5 models
- a score based on RMS and another based on correctly
  predicted residue-residue contacts (that's where the 4 A
  come in).
I personnally believe it's too late to change these rules.


Date: Thu, 13 Sep 2001 18:09:17 -0500
From: "Brian K. Shoichet"
Subject: Re: CAPRI scoring


Although I appreciate Julie's concerns, I basically agree with Joel and Shoshana, for what' it's worth. I am not worried about different criteria being played with, because I think that we are all on the same side and have the same interests. Also, I must say that here in Chicago we are in such confusion about these docking targets, and are so skeptical about the results that we are seeing, that we would be shocked and amazed to learn that including or not-including hydrogens would make a difference.

This may only reflect our own primativeness though.
All the best,

From Fri Sep 14 08:23:25 2001
Date: Thu, 13 Sep 2001 17:49:37 -0700
Subject: CAPRI scoring, pt. 2

Hello everyone,

This will be my second (and last) general message on the topic of a 4A cutoff. I have done some analysis on two crystal structures to determine the difference in residue contacts obtained with using hydrogen atoms vs no hydrogens and also for different cutoff radii.

For one of the two systems, the addition of hydrogens resulted in a whopping 28% increase in the number of residue contacts within 4A, and the other generated an increase of 18%. Increasing the cutoff to 5A (with no hydrogens) resulted in an increase of about 80% for both, which is perhaps too much. Increasing it to 4.5A gave a more modest increase of about 45%.

I respectfully disagree with those individuals who suggest that they do not expect this issue to make much of a difference because they do not expect the precictions to be very good in general. I think it is essential that the scoring mechanism be likely to return good scores for good solutions, regardless of whether we expect any. Since hydrogens were nowhere mentioned in the public literature for CAPRI, I do not think it is inappropriate to request a modification, even at this late date.

Anyone who is interested in examining the full results of the contact analysis can download the files at:

  • capscore.tar.gz
  • Capri copy

    Best wishes and happy docking,

    Julie Mitchell

    Date: Thu, 13 Sep 2001 11:07:08 -0400
    From: Carlos J. Camacho
    Subject: Re: CAPRI scoring

    Since it seems rather clear where ATP binds and also the phsopho. site in target 1, how different can 10 models be? One could send 10 models of more or less similar structures and if they are correct one could get a score of as much as 30+20+18+16+14 .. points for basicaly the same answer!

    It does not make much sense.

    I also think that If one cares about overall RMS, 4 A is certainly too small. Since one can trivialy improve p-p contacts by increasing overlaps in detriment of RMS.

    Personal opinion:

    I am sorry but I only now read the evaluation criteria.

    As a modeler I believe that RMS, as described in 2.4, is the best evaluation. And keeping with the spirit of moving to fully automated predictions is again the only one that makes sense since p-p contacts only benefit manual intervention.

    Also TEN models is too much. I believe that the score should be weighted by the number of models submitted. Such that people do not waste time analyzing "why not" models but only positive models. I also think that 10 different models are not very useful for an experimentalist, or proteomics.

    I do not think it is too late to put forward reasonable scoring methods. In fact my proposal is that there should not be a unique scoring method. We are here to learn from this competition, I don't care about a trophy, nor I think there should be one.

    All methods should be used to assess the models. And the winner should be on the eye of the beholder.

    Carlos J. Camacho

    Date: Sat, 15 Sep 2001 11:16:03 -0400 (EDT)
    From: Zhiping Weng
    Subject: Re: CAPRI scoring, pt. 2


    After viewing Julie's files, I agree with Julie: 4A is too strict and will only pull out excellent structures and won't discriminate good from bad ones. This will probably be important since the targets are all very difficult. I think we should be open to try multiple scoring criteria.


    Date: Mon, 17 Sep 2001 10:07:28 +0100 (BST)
    From: Michael Sternberg
    Subject: Re: possible solution?

    I agree with Joel, but have other comments.

    As CASP, the asessors nearly always had to develop the scoring criteria based on the actual results. Whilst the general rules need to be defined at adhered to in advance, details need to be altered.

    What is crucial is that the entries are blind to the assessors. If the evaluation is to run over a long period of time, this may be hard to maintain. If the results become unblinded, e.g. by publication of the coordinates, re defining criteria will be harder.

    One approach is for the assessors to have the results from this round and work out criteria which we then try to stick to over the timescale of the assessment.

    Another issue is whether groups need to submit every target or can be selective. At CASP, this has made assessment very hard. One cannot simply average over easy and hard targets, as groups that just submit easy targets will do better.

    It may be helpful for the assessors to discuss with the CASP organisers, obviously John Moult, but Tim Hubaard is local to the EBI and perhaps could help.


    Date: Sun, 16 Sep 2001 10:06:03 +0200 (MET DST)
    From: Joel Janin
    Subject: Re: possible solution?

    Dear Julie,

    that's the only reasonable thing to do. Neither Shoshana nor I intend to declare a winner! Shoshana is being asked to produce numbers that will be useful to all the participants, not to award ranks or prices. With only three targets, giving 'points' as Mike suggested is unnecessary. We can leave that to be discussed at a meeting. I bet other problems other than the 4 or 5 A cutoff will arise during Shoshana's work. They will have to be decided upon among the organizing committe, and then reported and discussed at the evaluation meeting.



    Date: Sun, 16 Sep 2001 03:05:44 -0700
    From: Julie Mitchell
    Subject: Re: possible solution?


    Thanks, I think this is the best thing. You might want to ask participants not to skew any of their results based on the criteria you have posted. I personally cooked up a utility to increase the number of contacts within a 4A cutoff, which I will avoid using so as not to bias your analysis.

    And I apologize for being somewhat difficult, but since we have arrived at a mutually agreeable conclusion, I think it was all for the best.