ArrayExpress

Programmatic access

1. REST-style queries to retrieve results in XML format
  1.1. Keyword searches for experiments and data files
  1.2. Specifing particular fields for searching
  1.3. Expanding searches using the Experimental Factor Ontology
  1.4. Construction of queries using AND, OR and NOT operators
  1.5. Filtering to get ArrayExpress direct submission data
  1.6. Filtering by counts
  1.7. Sorting
  1.8. Old-style queries
    1.8.1. Keyword searches
    1.8.2. Species searches
    1.8.3. Retrieving files
  1.9. Accessing private data
  1.10. Retrieve information about all public experiments
  1.11. Format of XML results
    1.11.1. Searches for experiments
    1.11.2. Searches for files
2. JSON web services format
3. Changes to programmatic access since September 2010

1. REST-style queries to retrieve results in XML format

Experiments and files linked to experiments can by searched for using keywords, by searching specific fields (e.g. sample attributes or experiment types), or by selecting experiments that have fulfill certain conditions such as the number of assays (hybridizations) or were released on a particular date.

1.1. Keyword searches for experiments and files associated with experiments

Keyword searches of all fields for experiments and files linked to experiments can be made using the following format of URLs:

A few points to note when using keyword search:

• Accession number and keyword searches are case insensitive
• Phrases of more than one word must be entered in quotes e.g. keywords="breast cancer cells"
• More than one keyword can by searched for using the '+' sign e.g. keywords=lung+cancer. The search treats these as 'AND' statements. See below for using OR and NOT.
• Use an asterisk * as a multiple character wild card e.g. keywords=colo* will search for colon, colorectal, color etc
• Use a question mark ? as a single character wild card e.g. keywords=te?t will search for text and test

Top

1.2 Specifing particular fields for searching

The following terms can be used to specify the field in which a term is searched for. Either experiments or files can be searched for in all cases by using the 'experiments' or 'files' term in the URL.

To link different search criteria together use the '&' symbol. E.g.

Top

1.3 Expanding searches using the Experimental Factor Ontology

The Experimental Factor Ontology is an application-focused ontology modelling the experimental factors in ArrayExpress. If the term expandefo=true is included in the URL then the search will be for occurrences of your term, and for synonyms and child terms in the Experimental Factor Ontology. E.g. if 'cancer' is entered the search will be for the term cancer, for synonyms of cancer and sub types of cancer such as lymphoma.

Some specific fields have EFO expansion on always: experimental factor value, experiment type, sample attribute and species.

Top

1.4 Construction of queries using AND, OR and NOT operators

More complex queries can be constructed using the operators AND, OR or NOT. AND is the default if no operator is specified. Again, either experiments or files can be searched for in all cases by using the 'experiments' or 'files' term in the URL.

Top

1.5 Filtering to get ArrayExpress direct submission data

Searches can be limited to experiments submitted directly to ArrayExpress (this is excludes data imported from the Gene Expression Omnibus (GEO, www.ncbi.nlm.nih.gov/geo/)) or to only the imported data. To do this use the following syntax:

For more information about how we import data from GEO see the GEO data help page.

Top

1.6 Filtering by counts

Experiments fulfilling certain count critera can also be searched for e.g. having more than 10 assays (hybridizations). These searches use the following syntax:

Top

1.7 Sorting

The results of a query can be sorted on several fields in ascending or descending order using sortby=xxx and sortorder=ascending/descending. The fields that can be used for sorting are:

• accession
• name
• assays
• species
• releasedate
• fgem (for "final gene expression matrix", i.e. processed data)
  
• raw
• atlas

Example queries:

Top

1.8 Old-style queries

The URL format for ArrayExpress searches was changed in September 2010 to allow searching of specific fields, use of AND, OR, NOT, counts of attributes and ordering of results.

The old-style URLs are still fully functional however. The format of the pre-September 2010 URL is described below.

 Top

1.8.1. Keyword searches

Keywords can be used to search for specific experiments, with the results returned in an XML document, using URLs with the experiments term and keywords=X format. E.g.

To narrow the search by using more than one keyword, separate terms by + e.g.

Top

1.8.2 Species searches

To narrow search to experiments with samples of a particular species include the 'species' term

op

1.8.3 Retrieving files

To retrieve the list of files associated with a set of experiments use the 'files' term in the URL

To retrieve a list of files for a particular experiment format the query as follows:

Also note:

• Accession number and keyword searches are case insensitive.
• Parts of words can also be searched e.g. colo will retrieve results with matches to both colon and colorectal (and also color, colony etc), unless there is a paramter wholewords=on in the query string e.g.

Top

1.9 Accessing private data

Private data are usually pre-published/unpublished data. Access to private data is under password control.

1. Use your client to retrieve the following URL (inserting the username and password provided to you by the ArrayExpress curators):
   http://www.ebi.ac.uk/arrayexpress/verify-login.txt?u=username&p=password If the login details are correct this will return a login token that is unique to your username, IP address and client. If the login fails you will get a blank page.
2. Set up 2 cookies for the domain http://www.ebi.ac.uk/ :
   AeLoggedUser = username
   AeLoginToken = login token from URL in step 1
3. Use these cookies when making all subsequent requests to ArrayExpress. It is important to use the same client that was used in step 1.

Top

1.10 Retrieve information about all public experiments

To retrieve an xml file with information about all public experiments remove the keyword part of the search.

Top

1.11 Format of XML results

The XML documents returned from a search lists how many experiments were retrieved and then either information about the experiment or lists the files associated with an experiment depending on the search made. In both cases information about each experiment is in an <experiment></experiment> element.

Top

1.11.1 Searches for experiments

Example:

Top

1.11.2 Searches for files

Example:

Top

2. JSON web services format

To retrieve the results of queries in JSON format, the base URL changes from http://www.ebi.ac.uk/arrayexpress/xml/v2/ to http://www.ebi.ac.uk/arrayexpress/json/v2/. All queries described above can be carried out and will produce a JSON-format file while can by downloaded.

E.g. experiment queries:

E.g. file queries:

There is one extra parameter 'jsonp' - Enable JSONP; the JSON output will be prepended with the value of the jsonp parameter and wrapped in parentheses. For example:

3. Changes to programmatic access since September 2010

See also section 1.8 above for old-style queries used before September 2010.

1. URL change - the base URL for queries has changed to http://www.ebi.ac.uk/arrayexpress/xml/v2/. The previous format for queries is still supported.
2. Additional functionality including filtering and complex queries (AND, OR , NOT) has been added
3. Addition of JSON web services