EGA Webin Data Uploader

EGA Webin Data Uploader

This tool enables submitters to encrypt, generate md5sums and upload each file to their submission account using FTP. The tool can be run as a JAVA client using a graphical interface or from the command line, enabling pipeline incorporation and greater flexibilty.  For example, to encrypt and generate md5sums for each file for upload using Aspera, should FTP be deemed too slow for long distance transfers.  

How to install

Using the graphical interface

Using from the command line

How to encrypt & create md5sum's for upload with Aspera (Useful for large data files)

Troubleshooting

 

How to install

Please note the EGA Webin Data Uploader requires Java JRE 6  or OpenJDK.

To enable file encryption, unlimited strength JCE policy files must be installed if you are using JAVA JRE.

 

Installing the JCE policy files:

  • Download the unlimited strength JCE policy files (JRE 6 / JRE 7
  • Uncompress and extract the downloaded file.

    This will create a subdirectory called JCE. This directory contains the following files:

            => README.txt

            => COPYRIGHT.html

            => local_policy.jar

            => US_export_policy.jar

  • Install the two policy JAR files by replacing the existing ones in your <java-   home> directory.

    The standard place for JCE jurisdiction policy JAR files is:

            => <java-home>/lib/security [Unix]

            => <java-home>\lib\security [Win32]

Notes:

<java-home> refers to the directory where the Java SE Runtime Environment (JRE) was installed.

 

 How to use graphical interface

  • Launch the EGA Webin Data Uploader.
  • Accept to download the EGA-WebinDataUploader.jnlp file.
  • Launch the EGA-WebinDataUploader.jnlp file and confirm you wish to run the application.
  • Enter your submission account username and password.
  • Choose upload protocol: FTP or UDP (if availiable for your network)
  • Choose the source directory of the files you wish to upload (Figure 1).
  • Select individual files for upload or use the 'Select All' button to select all files for upload (Figure 2).

Figure 1: Choosing your source directory

 

Figure 2: Selecting files for upload

  • Choose 'Override Existing Files' for replacing existing files of the same filename.
  • Choose 'Upload Directory Tree' option if you wish to maintain your directory structure in your submission account. If this option is not selected, files will be uploaded to the root directory of your submission account.
  • Click on the 'Upload' button.  Files will be encrypted, md5sums generated and uploaded to your submission account.
  • Right click on the 'Upload' button to enter 'debug mode' to capture extra information about upload errors.

The uploader writes files to the source directory of your local file system (MD5 files, log files and BAM index files), as a result you must have write-access permissions for the source directory.  

 

How to use from the command line

java -jar webin-data-streamer-Upload-Client.jar -p -user -pass -files

Or for parallel uploads:

java -jar webin-data-streamer-Upload-Client.jar -pp <n> -user -pass -files

 

{where <n> is the number of parallel upload threads}

'*' may be used to upload all files in a directory.

 

Parameters:       

-user <username>

-pass <password>-fm {"file mode" - process everything to file instead of FTP; username, password not required in this case}

-v {"verbose"}

-vv {"very verbose"}

<>-o {"overwrite" files on the server - otherwise files are not uploaded if they already exist on the server}

-index {creates an index file for BAM files, if possible, and uploads it as well}

-udt {uses UDT protocol, if available. This cancels parallel uploads, for the time being}

-f, -file, -files <filepath(s)> {a space-separated list of 1-many files to be uploaded.}

Please provide the path to the webin-data-streamer-Upload-Client.jar and run the command from within the directory the file/s are located.

The uploader writes files to the source directory of your local file system (MD5 files, log files and BAM index files), as a result you must have write-access permissions for the source directory.  

 

How to encrypt and create md5sum's for upload with Aspera (Useful for large data files) 

  java -jar ../webin-data-streamer-Upload-Client.jar -file file1 file2

         '*' may be used to refer to all files in source directory.

       The output files (3 files per input file: encrypted files, plain-file-md5 file, and encrypted-file-md5 file) are created in the same directory as the original file

  •  All output files should then be uploaded to your submission account using Aspera or FTP.    

 

Troubleshooting

 

I cannot log-in to the EGA Webin Data Uploader, what should I do?

The error message displayed should be 'Login Error' (or 'Login with USER first' or 'Permission denied').

The Uploader logs directly in to the ENA FTP server. If there are errors during login, they should be replicable using a different client, such as FileZilla.

The reason could be due to incorrect user credentials (e.g. account not created yet, typos, etc.) or may be a firewall issue.

 Solution 1: (1) Type username and password (don't use copy-and-paste). (2) Replicate the error using FileZilla (3) Contact the EGA Helpdesk for further assistance.
 Solution 2: (1) Change your ports used by TCP connections to allow port range of 8121-8210 (2) Your firewall should enable access to the following IP addresses: 193.62.193.198, 193.62.193.197 and 193.62.193.198 3) If necessary ask your systems department to make these changes
 
I have an 'MD5 checksum not calculated' error message when trying to upload my files with the EGA Webin Data Uploader, what can I do?

This is a status message that is displayed if no MD5 checksum was calculated. It is not an error message that indicates that an error occured during the calculation of the MD5 checksum.

The reason an MD5 checksum is not calculated is that a file was not successfully uploaded.

There are multiple possibilities why a file was not uploaded. In almost all cases the reason is a local firewall, which prevents the Uploader from establishing a data connection. The EGA-WebinDataUploader uses passive FTP data channels for uploads, which are sometimes blocked by local firewalls in default configuration. Other FTP clients, such as FileZilla, use active FTP data channels, which are allowed by default firewalls.

Solution: contact your local systems administrator to allow passive FTP data channels.

 
Upload is failing a while after the upload has started with EGA Webin Data Uploader, why is that?

Some firewalls react if unusual behavior is detected; uploading large files using passive FTP data channels might be interpreted that way.

Solution: contact your local systems administrator to allow passive FTP data channels.

 

 
"FTP list throwable (0): Connection reset {file} IOException: java.net.SocketException: Connection reset by peer: socket write error Upload of 0 file(s) Completed. MD5 issues!"

The list command is first run to check if the file to be uploaded already exists on the server. This is the first time an FTP data connection is used (to transfer the directory content of the server). If a firewall interferes with the data connection then this error will be produced.

Solution: contact your local systems administrator to allow passive FTP data channels.

 

 
When running from the command line, I observe the following error: UnixFileSystem.createFileExclusively (Native Method)

The error is thrown by UNIX (“UnixFileSystem.createFileExclusively(Native Method)”).  It appears that the user does not have write-access to the file system where the file to be uploaded is located.

The Uploader always writes MD5 checksums into files before uploading them to the server, and these files are created in the same location where the uploaded file itself resides.

Solution: address your directory permission issue and re-run the command.