Scoring Discussion

Date: Tue, 11 Sep 2001 13:38:27 +0100 (BST)
From: Graham Smith smithgr@icrf.icnet.uk
Subject: capri submission



Dear Kim,

I just wanted to check a few things about the submission format
before finalizing my submissions. I see from the submissions page
that a script will automatically check the complexes but I would 
like to check what's going on before I submit them as it might
otherwise lead to a scramble to meet the deadline! 

In particular

Do I have to resubmit the haemagglutinin antibody (3rd target)  as it
came, with atom numbers starting from 11680, and a strange atom
order within residues? 

The kinase in the 1st target came with some sidechains missing (not
resolved), which I built on before docking. Do I have to remove them
again? 

The substrate in the 1st target was a S46D mutant, which I mutated 
back to S46 before docking. Do I have to put it back as D? (I imagine the
answer to this one at least is yes)


Thanks, 

Regards, 

-Graham 





From henrick@ebi.ac.uk Fri Sep 14 08:26:40 2001
Date: Tue, 11 Sep 2001 14:21:34 +0100 (BST)
Subject: Re: capri submission



Dear Graham,

re the targets - these were all as supplied by Joel
and put into chain ID order by us at the EBI

Certainly the atom serial number is of no consequence and
I cannot see this being used in any way.

as far as the atom order goes all submissions are processed to a standard
pdb style before we are send them on to the accessors
e.g. traget 03 will what ever order given and whether it will have H's
will become


ATOM  15588  N   GLN L   1 
ATOM  15589  CA  GLN L   1 
ATOM  15590  C   GLN L   1 
ATOM  15591  O   GLN L   1 
ATOM  15592  CB  GLN L   1 
ATOM  15593  CG  GLN L   1 
ATOM  15594  CD  GLN L   1 
ATOM  15595  OE1 GLN L   1 
ATOM  15597  NE2 GLN L   1 


(from the supplied target file of



ATOM  11680  CB  GLN L   1
ATOM  11681  CG  GLN L   1
ATOM  11682  CD  GLN L   1
ATOM  11683  OE1 GLN L   1
ATOM  11684  NE2 GLN L   1
ATOM  11685  C   GLN L   1
ATOM  11686  O   GLN L   1
ATOM  11687  N   GLN L   1
ATOM  11688  CA  GLN L   1  


which also has the occupancy in wrong columns)

re target 01

 modelling in of complete side chains is a plus I would think
 the target set minus some side chains is the observed density

re the S46D mutant - as far as I understood from Joel
   the coordinates supplied were for what was the actual
   structures that have been determined and  Ser46 is rather
   crucial to the mechanism but the ASP is the un-published 
   structure in 1JB1 (un-complexed) - I dont have infomation
   if the complex is the Ser46Asp mutant or the native Ser46
   The target docked complex is not specially noted to have ADP
   and I imagine the ASP mutant is so one can isolate the ADP
   in the crystal - similarly the target is supplied as the Seleno Met
   and the docked target is likely to molecular replacement so
   would be the MET rather than MSE, as HPr-Ser46 is suggested
   to have key importance in the interface with the kinase
   then the target protein may well be the Ser46 and may play
   an important role in the docking

   These differences Ser46/Asp46 and MSE/MET however should not
   in my opinion be significant in the Assessment

   I suggest you leave it as Ser


as far as the deadline - we will probably be open for 
submissions for a couple of days after the 16th

kim




Date: Tue, 11 Sep 2001 15:04:33 +0100 (BST)
From: Graham Smith smithgr@icrf.icnet.uk
Subject: Re: capri submission


Dear Kim, 

Thanks for the message and the help. (I had forgotten about the MSE
residues, which I had changed to MET). I'll put in the submissions 
along the lines you suggest. 


Thanks again, 

-Graham 





From: "E.J.Gardiner" E.Gardiner@sheffield.ac.uk
Date: Tue, 11 Sep 2001 09:30:21 +0100
Subject: Capri submission



Dear Kim

Apologies if you have already replied to a similar e-amil - if so I 
haven't received it.
In docking target 1, we have used A different HPR, pdb code 1ptf, 
rather than 1sph as supplied by the contest. Do you wish us to 
superpose 1sph onto our models? It will (hopefully) make no to the 
answer, but might different residues throw an automatic checking 
procedure?


Thanks
Eelanor
Dr Eleanor Gardiner  





Date: Tue, 11 Sep 2001 09:52:42 +0100 (BST)
From: Kim Henrick 
Subject: Re: Capri submission


Dear Eleanor,

In Joel's original note he did say one could
use chains from different pdb entries to those
supplied in the target files - so this should
be allowed. However for our automatic first
stage checking procedures to get the submission
into a standard form that should make subsequent
accessments easier we assumed that people would
start from the target xyz files on the capri
web pages - so that what ever accessment was 
to be carried out after we at the ebi pass on
the files to the accessors would be more suitable
to an automatic protocol.

I havent checked but is the sequence for 1ptf the same
as 1sph ? - the main thing would be to
get the chain ID's to match and the model records
included,  then as Joel allowed this in the first
announcement then I assume that the assessors
have thought of this and have a procedure that 
will be able to cope with what you have.

kim




Date: Thu, 13 Sep 2001 04:02:05 -0700
From: Miri Eisenstein miri@wildtype.weizmann.ac.il
Subject: Re: CAPRI scoring



Hello everybody,

It is too late, in my opinion, to change the mode of evaluation and
scoring for the current CAPRI experiment. So, the predictions should be
evaluated with a cutoff distance of 4 Angstroms, for better for worse.
However, Shoshana's suggestion to test other distance cutoffs and the
sensitivity of the results to different cutoffs should be favorably
considered. It will help us, the predictors, and will provide information
for the future CAPRI's.

Miri





Date: Thu, 13 Sep 2001 01:02:07 -0700
From: Julie Mitchell mitchell@sdsc.edu
Subject: Re: CAPRI scoring question


Joel,

Thanks for the prompt response.  I personally think that in
the absence of hydrogens that a 4 Angstrom cutoff for residue
contacts is about 1 Angstrom too small.  It effectively 
encourages participants to submit structures that would have
collisions when modeled properly with all the hydrogens.

As a minor side note, I think the name "Critial Assessment
of Predicted Interactions" reads a little better than 
"Critical Assessment of Prediction of Interactions"

With kind regards,


Julie




 
Date: Thu, 13 Sep 2001 09:56:21 +0200 (MET DST)
From: Joel Janin janin@lebs.cnrs-gif.fr
Subject: Re: CAPRI scoring question


Dear Julie,

'our' criterion, as you cite it, is taken from M. Sternberg's
proposal. As none of the participants has challenged it, I expect
Shoshana will use it as it stands - ignoring  hydrogens. I do not 
expect the predictions to reach the level of accuracy where it
makes a difference! 

Yours, 

Joel



On Wed, 12 Sep 2001, Julie Mitchell wrote:

 
 Shoshana and Joel,
 
 I have a simple question regarding the scoring of residue 
 contacts for the CAPRI targets.  As I understand your 
 criterion, two residues are deemed in contact provided 
 there is a pair of atoms (one from each residue) that are
 within 4 Angstoms distance of one another.  My question 
 is whether hydrogen atoms are taken into account or whether
 only hard atoms are used in determining residue contacts.
 
 Thanks in advance,

 
 Julie Mitchell

 


Date: Thu, 13 Sep 2001 01:59:21 -0700
From: Julie Mitchell mitchell@sdsc.edu
Subject: CAPRI scoring


Dear fellow CAPRI participants,

An issue has come up regarding the scoring of residue contacts
for CAPRI, and Joel Janin has suggested I circulate something
to all the participants.  

As you are likely aware, the scoring of predictions is based
on residue contacts.  The question I asked of the organizing
committee is whether hydrogen atoms will be used in determining
residue contacts.  The response I got back was that only
non-hydrogen atoms will be used.  My opinion is that a 4
Angstrom cutoff in the absence of hydrogens is too small, and
I suggest that 5 Angstroms is a more reasonable cutoff in 
this case.  

Shoshana has just now suggested that they intend to 
review how the results change as the cutoff is varied.
I do, however, think that changing the scoring criteria 
midstream will cause arguments and that it is best to have a
specific cutoff going in that people agree on.

If you have an opinion on this issue, please forward your
comments to Joel Janin janin@lebs.cnrs-gif.fr and Shoshana
Wodak shosh@ucmb.ulb.ac.be

Best wishes (and good luck with your predictions),


Julie Mitchell

PS: I am appending a copy of email correspondence
with Joel and Shoshana


Joel Janin wrote:

  Joel,
 
  Thanks for the prompt response.  I personally think that in
  the absence of hydrogens that a 4 Angstrom cutoff for residue
  contacts is about 1 Angstrom too small.  It effectively
  encourages participants to submit structures that would have
  collisions when modeled properly with all the hydrogens.
 
         You are welcome to circulate the suggestion to
         other participants.
 
  As a minor side note, I think the name "Critial Assessment
  of Predicted Interactions" reads a little better than
  "Critical Assessment of Prediction of Interactions"
 
         I fully suscribe to that suggestion.

 
  With kind regards,

 
  Julie



Shoshana Wodak wrote:

 
    Julie,
 
    Dont worry. In any case we would have tested several cutoff, just to see
    how results change. But you are welcome to make your proposal.

 
       Shoshana

 
 I just rechecked the website, and hydrogens aren't mentioned
 anywhere in the documents.  I was pretty sure I'd read them
 carefully before.  Thanks again for clarifying the details
 of the organizing committee's criteria.
 
 -Julie

 
 Shoshana Wodak wrote:

 
      Dear Julie,
 
      Given the accuracy of most calculations, it seems reasonable not
to consider hyrdogen atoms, as stated in the original proposal for the
evaluation.

 
         Shoshanan




Date: Thu, 13 Sep 2001 18:06:34 +0200 (MET DST)
From: Joel Janin janin@lebs.cnrs-gif.fr
Subject: Re: scoring



Dear Elizabeth,


the rules were up to now:
- 5 models
- a score based on RMS and another based on correctly
  predicted residue-residue contacts (that's where the 4 A
  come in).
I personnally believe it's too late to change these rules.


Yours,


Joel






Date: Thu, 13 Sep 2001 18:09:17 -0500
From: "Brian K. Shoichet" shoichet@model.pharm.northwestern.edu
Subject: Re: CAPRI scoring


Hi,

Although I appreciate Julie's concerns, I basically agree with Joel and
Shoshana, for what' it's worth.  I am not worried about different criteria
being played with, because I think that we are all on the same side and
have the same interests.  Also, I must say that here in Chicago we are in
such confusion about these docking targets, and are so skeptical about the
results that we are seeing, that we would be shocked and amazed to learn
that including or not-including hydrogens would make a difference.

This may only reflect our own primativeness though.  


All the best,


Brian




From mitchell@sdsc.edu Fri Sep 14 08:23:25 2001
Date: Thu, 13 Sep 2001 17:49:37 -0700
Subject: CAPRI scoring, pt. 2



Hello everyone,

This will be my second (and last) general message on the topic of 
a 4A cutoff.  I have done some analysis on two crystal structures
to determine the difference in residue contacts obtained with using
hydrogen atoms vs no hydrogens and also for different cutoff radii.

For one of the two systems, the addition of hydrogens resulted in 
a whopping 28% increase in the number of residue contacts within
4A, and the other generated an increase of 18%.  Increasing the cutoff
to 5A (with no hydrogens) resulted in an increase of about 80% for 
both, which is perhaps too much.  Increasing it to 4.5A gave a more 
modest increase of about 45%.

I respectfully disagree with those individuals who suggest that they
do not expect this issue to make much of a difference because they
do not expect the precictions to be very good in general.  I think it 
is essential that the scoring mechanism be likely to return good 
scores for good solutions, regardless of whether we expect any.  Since 
hydrogens were nowhere mentioned in the public literature for CAPRI, 
I do not think it is inappropriate to request a modification, even at 
this late date.

Anyone who is interested in examining the full results of the contact
analysis can download the files at:


capscore.tar.gz
Capri copy 

Best wishes and happy docking,

Julie Mitchell




Date: Thu, 13 Sep 2001 11:07:08 -0400
From: Carlos J. Camacho ccamacho@tonka.bu.edu
Subject: Re: CAPRI scoring


Since it seems rather clear where ATP binds and also the phsopho. site
in target 1, how different can 10 models be? One could send 10 models of
more or less similar structures
and if they are correct one could get a score of as much as
30+20+18+16+14 .. points for basicaly the same answer!

It does not make much sense. 

I also think that If one cares about overall RMS, 4 A is certainly too
small. Since one can trivialy improve  p-p contacts  by
increasing overlaps in detriment of RMS.

Personal opinion:

I am sorry but I only now read the evaluation criteria.

As a modeler I believe that RMS, as described in 2.4, is 
the best evaluation. And keeping with the spirit of moving
to fully automated predictions is again the only one that
makes sense since p-p contacts only benefit manual 
intervention. 

Also TEN models is too much. I believe that the score should
be weighted by the number of models submitted. Such that people
do not waste time analyzing "why not" models but only positive
models. I also think that 10 different models are not very 
useful for an experimentalist, or proteomics.

I do not think it is too late to put forward reasonable 
scoring methods. In fact my proposal is that there should not be
a unique scoring method. We are here to learn from this
competition, I don't care about a trophy, nor I think there 
should be one.

All methods should be used to assess the models. And the 
winner should be on the eye of the beholder. 

Carlos J. Camacho





Date: Sat, 15 Sep 2001 11:16:03 -0400 (EDT)
From: Zhiping Weng zhiping@bu.edu
Subject: Re: CAPRI scoring, pt. 2


Hello

After viewing Julie's files, I agree with Julie: 4A is too strict and will
only pull out excellent structures and won't discriminate good from bad
ones.  This will probably be important since the targets are all very
difficult.  I think we should be open to try multiple scoring criteria.

Zhiping





Date: Mon, 17 Sep 2001 10:07:28 +0100 (BST)
From: Michael Sternberg sternber@icrf.icnet.uk
Subject: Re: possible solution?


I agree with Joel, but have other comments.

As CASP, the asessors nearly always had to develop the scoring criteria
based on the actual results.   Whilst the general rules need to be
defined at adhered to in advance, details need to be altered.

What is crucial is that the entries are blind to the assessors.
If the evaluation is to run over a long period of time,
this may be hard to maintain.  If the results become unblinded,
e.g. by publication of the coordinates, re defining
criteria will be harder.

One approach is for the assessors to have the results from this 
round and work out criteria which we then try to stick to over the 
timescale of the assessment.

Another issue is whether groups need to submit every target
or can be selective.   At CASP, this has made assessment very hard.
One cannot simply average over easy and hard targets,
as groups that just submit easy targets will do better.

It may be helpful for the assessors to discuss with the CASP organisers,
obviously John Moult, but Tim Hubaard is local to the EBI
and perhaps could help.


Mike





Date: Sun, 16 Sep 2001 10:06:03 +0200 (MET DST)
From: Joel Janin janin@lebs.cnrs-gif.fr
Subject: Re: possible solution?



Dear Julie,

that's the only reasonable thing to do. 
Neither Shoshana nor I intend to declare a winner! 
Shoshana is being asked to produce numbers that will be useful to 
all the participants, not to award ranks or prices. With only
three targets, giving 'points' as Mike suggested is unnecessary.
We can leave that to be discussed at a meeting.
I bet other problems other than the 4 or 5 A cutoff will arise
during Shoshana's work. They will have to be decided upon among the
organizing committe, and then reported and discussed at the
evaluation meeting.

Yours,

Joel





Date: Sun, 16 Sep 2001 03:05:44 -0700
From: Julie Mitchell mitchell@sdsc.edu
Subject: Re: possible solution?


Joel,

Thanks, I think this is the best thing.  You might want to
ask participants not to skew any of their results based on
the criteria you have posted.  I personally cooked up a
utility to increase the number of contacts within a 4A cutoff,
which I will avoid using so as not to bias your analysis.

And I apologize for being somewhat difficult, but since we 
have arrived at a mutually agreeable conclusion, I think it
was all for the best.

Julie