 |
ArrayExpress Developer Information
Software developed by the microarray group can be installed and run locally. A list of available software can be found here:
ArrayExpress and other software >>
There are statistics on the content of the ArrayExpress archive and Gene Expression Atlas here:
ArrayExpress statistics >>
Developer and Local User FAQ
- Can I set up a local version of ArrayExpress?
- I am considering installing ArrayExpress locally, and I wonder what are the minimum requirements for the hardware and whether it will work under Linux and Oracle9i?
- Is it possible to get direct access to the ArrayExpress database?
- Do you have a relational ER diagram for the database schema?
- An Oracle sequence "SEQ" is used to generate primary key values for all tables in ArrayExpress. MySQL doesn't have sequences, but it has auto increment for primary key columns of each table. Can I use auto increment to generate primary key values for each table individually, or does ArrayExpress implementation require unique PK values across all tables?
- In mapping.xml file for Castor JDO, key-generator is declared, is it used?
- We are trying to compile your source code, but don't know what to do with files in MAGE/manual subdirectory with .mn and .im extensions.
- I have some developer questions not answered here, who do I ask?
- I want to load only partial MAGE-ML into the database, e.g., only biosequence package or arraydesign package; is that possible?
- We are trying to load some Affymetrix data, and the loader crashes with "Out of memory" exception. Are there any workarounds?
- I get the following error message when I load the data, what is the problem?
===============================================
Error while creating pool of Databases
java.lang.NullPointerException
at DBInterface.DBPool.(DBPool.java:100)
at DBInterface.DBPool.create(DBPool.java:237)
at MAGEloader.SpecializedLoad.init
(SpecializedLoad.java:118)
at MAGEloader.SpecializedLoad.doLoad
(SpecializedLoad.java:162)
at MAGEloader.MAGEdriver.main
(MAGEdriver.java:441)
......
Developer FAQ
- Can I set up a local version of the ArrayExpress Archive?
Please refer to this software web page. Note that we are redeveloping the ArrayExpress infrastructure and suggest you contact us at arrayexpress@ebi.ac.ukbefore deploying this software
- I am considering installing ArrayExpress locally, and I wonder what are the minimum requirements for the hardware and whether it will work under Linux and Oracle9i.
We are running ArrayExpress on Oracle 9i. The OS is Compaq True Unix, but it's it should work on linux as all supporting software is in Java. Regarding hardware requirements, it's hard to say what the minimum is. We have two 4-processor Compaq Alfa servers, 8G memory each, one is used as an Oracle server and the other as an application server, but everything should work also on smaller machines; we run data validation and loading applications with 1.5Gb memory limit, and our production Tomcat for data access servlets with 2Gb memory limit, however another Tomcat instance works with 1Gb. We are redeveloping the ArrayExpress infrastructure and therefore we recommend you contact us for advice before attempting to deploy ArrayExpress components.
- Is it possible to get direct access to the ArrayExpress database?
At the moment we provide access via our web-interface and programmatic access is available. We do not provide direct SQL access for security reasons.
- Do you have a relational ER diagram for the database schema?
Since the ArrayExpress schema is derived from MAGE-OM and it is quite large and complex, we don't have a visual representation. If you need to understand the schema, we suggest you to learn the basics of MAGE, and it's possible to see the principles how the schema is derived from the model. For more information, please refer to this web page
- An Oracle sequence "SEQ" is used to generate primary key values for all tables in ArrayExpress. MySQL doesn't have sequences, but it has autoincrement for primary key columns of each table. Can I use autoincrement to generate primary key values for each table individually, or does ArrayExpress implementation require unique PK values across all tables?
Although unique PK values across all tables are not required, it is advisable to modify getCurrSeqVal(), getNextSeqVal(), DBInsertWithKey() and DBClobInsertWithKey() methods of DBInterface class, generating unique long values in some other manner. In order to take advantage of the autoincrement feature of MySQL you would need to make more modifications.
- In mapping.xml file for Castor JDO, key-generator is declared, is it used?
No, in ArrayExpress Castor is used only for reading, not writing, therefore key-generator is not used.
- We are trying to compile your source code, but don't know what to do with files in MAGE/manual subdirectory with .mn and .im extensions.
These are files used by our code generator that generates most of the MAGE package classes (corresponding to MAGE-OM). All non-standard (i.e., non-generated) behavior is included in those files, but you don't have to do anything with them, contents of those files are already included in the corresponding MAGE package files.
- I have some developer questions not answered here, who do I ask?
Please contact us at arrayexpress@ebi.ac.uk")
.
Local User FAQ
- I want to load only partial MAGE-ML into the database, e.g., only biosequence package or arraydesign package; is that possible?
Yes, as long as your MAGE-ML file is valid, you can load it. The loader tries to resolve all <abc_ref identifier="xyz"> elements, first of all trying to match it with in the file (or set of files) being loaded, and then, if it can't, against data already in the database. We recommend running MAGEvalidator before loading; it will report all references that cannot be resolved, as well as all references that cannot be resolved locally but only against the database.
- We are trying to load some Affymetrix data, and the loader crashes with "Out of memory" exception. Are there any workarounds?
If multiple Affy chips are used and .cel level data is provided, several million feature identifiers are stored in memory. This can cause "Out of memory exception". To fix this problem here we always load ArrayDesigns first, check the ids against it using the validator, and then use a local parameter that tells the loader not to store Feature references in memory and resolve against the database later, but to make links immediately. If there is a file LocalProperties.txt in the load directory and it contains a line immediateFeatureRefResolve=true it prevents all the Feature id's being accumulated in memory.
- I get the following error message when I load the data, what is the problem?
==============================================
Error while creating pool of Databases
java.lang.NullPointerException
at DBInterface.DBPool.(DBPool.java:100)
at DBInterface.DBPool.create(DBPool.java:237)
at MAGEloader.SpecializedLoad.init
(SpecializedLoad.java:118)
at MAGEloader.SpecializedLoad.doLoad
(SpecializedLoad.java:162)
at MAGEloader.MAGEdriver.main
(MAGEdriver.java:441)
......
The cause of the error is database.xml file, present in the MAGE directory of your installation. You must edit the following attributes of the driver element of the database.xml file. url="jdbc:oracle:thin:@host:port:sid"> <param name="user" value="xxxxx" /> <param name="password" value="xxxxx" /> Here host, port, sid, user and password need to be changed according to your environment. There is some more important information in the README.txt file, it's worth it going through that before using the MAGEvalidator/MAGEloader.
- I get the following error message when when I run the MAGEvalidator e.g.:
========================================
Exception in thread "main"
java.lang.NoClassDefFoundError:
org/xml/sax/helpers/DefaultHandler
at java.lang.ClassLoader.defineClass0
(Native Method)
at java.lang.ClassLoader.defineClass
(ClassLoader.java:496)
at java.security.SecureClassLoader.
defineClass(SecureClassLoader.java:117)
at java.net.URLClassLoader.defineClass
(URLClassLoader.java:251)
at java.net.URLClassLoader.access$300
(URLClassLoader.java:69)
at java.net.URLClassLoader$ClassFinder.
run(URLClassLoader.java:544)
at java.security.AccessController.
doPrivileged(Native Method)
at java.net.URLClassLoader.findClass
(URLClassLoader.java:203)
at java.lang.ClassLoader.loadClass
(ClassLoader.java:325)
at sun.misc.Launcher$AppClassLoader.
loadClass(Launcher.java:380)
at java.lang.ClassLoader.loadClass
(ClassLoader.java:257)
at mageui.ConsoleMenu.displayOptions
(ConsoleMenu.java:204)
at mageui.ConsoleMenu.main
(ConsoleMenu.java:64)
......
This is failing to obtain the class files needed to execute the program. This may happen when you try to run the program from any other location other than the installation directory. So please run the script from the location where it is installed. See the readme for full details.
ftp://ftp.ebi.ac.uk/pub/databases/microarray/code/loader-validator/
- I have some developer questions not answered here, who do I ask?
Please contact us at arrayexpress@ebi.ac.uk
|