How to upload data to PRIDE via Aspera command line

Assuming you have consulted our Submission Guidelines and wish to upload your datasets via the Aspera command line, please follow the steps below.

Requirements: Please download the Aspera Connect Web Browser Plug-in. Although you download a Browser Plug-in you will be using the 'ascp' command line transfer program distributed with it. That means command line skills are needed in order to use the Aspera upload option.

  • Operating System: Windows XP / 2003 / Vista / 2008 / 7 / 8, Mac OS Intel 10.5 / 10.6 / 10.7 / 10.8 / Linux

You don't have to register in order to download the Browser Plug-in and the download is free of charge.

Check the command line transfer usage for more configuration details.

The location of the 'ascp' program in the filesystem:

Mac: on the desktop go cd /Applications/Aspera\ Connect.app/Contents/Resources/ there you'll see the command line utilities where you're going to use 'ascp'.

Windows: the downloaded files are a bit hidden. For instance in Windows 7 the ascp.exe is located in the users home directory in: AppData\Local\Programs\Aspera\Aspera Connect\bin\ascp.exe

Linux: should be in your user's home directory, cd /home/username/.aspera/connect/bin/ there you'll see the command line utilities where you're going to use 'ascp'.

Uploading a directory of files:

1. Asking PRIDE Support for target directory and password

The PRIDE Curators will specify a target directory for you, see <name-of-target-dir-specified-by-PRIDE> in the following commands, and you will be provided with this information and a password when you email PRIDE Support at pride-support@ebi.ac.uk

2 The upload command and process

When preparing your dataset please be sure to unambiguously assign a unique file name to all of your files. Please also upload the submission summary file into the same folder.

Mac or Linux: ./ascp -QT -l500m --file-manifest=text -k 2 -o Overwrite=diff <path-to-folder-to-be-uploaded> pride-drop-006@ah01.ebi.ac.uk:<name-of-target-dir-specified-by-PRIDE>

Windows: ascp.exe -QT -l500m --file-manifest=text -k 2 -o Overwrite=diff <path-to-folder-to-be-uploaded> pride-drop-006@ah01.ebi.ac.uk:<name-of-target-dir-specified-by-PRIDE>

The <path-to-folder-to-be-uploaded> should not have any blank spaces in it.

Please set the '--file-manifest=text -k 2' flags as well.

This will generate an aspera progress file on your side that will contain a report on the files that have been uploaded, also you can interrupt the process and then it will only upload the ones that were not there so no more overwriting files. It will also skip the ones that are already in the target directory.

If -l500m ~ 500 Mb/s is unstable and leads to timeouts then I suggest to go back to -l250m as the maximum transfer rate, even that is fast enough to transfer theoretically 2 TBs within a day.

Once upload has been finished you will be prompted to enter the password provided earlier.

3. Notify the PRIDE Team 

Email pride-support@ebi.ac.uk in case your upload has been succesfully finished.

Troubleshooting

If you are having problems with Aspera connection time-outs, it can be down to either one of the following.

  • Transfers cannot start - the connection fails instantly.

Ensure that TCP traffic on port 33001 is allowed (open) for outbound connections through your computer's firewall and network's firewall.

  • The connection is made, transfers are started, but 0 bytes (0%) are uploaded for each file.

Ensure that UDP traffic on port 33001 is allowed (open) for outbound connections through your computer's firewall and network's firewall.

 

Detailed Windows instructions

1. In your favourite web browser, go to: http://downloads.asperasoft.com/connect2/
and install the "Aspera Connect" web browser plug-in.

2. After you have successfully installed Aspera Connect, you need to locate the program itself, "ascp.exe".
This will live somewhere in your user home directory in Windows, e.g.:
C:\Users\<your user name\AppData\Local\Programs\Aspera\Aspera Connect\bin\

e.g. my directory on my Windows PC is (Figure 1):
"C:\Users\tobias\AppData\Local\Programs\Aspera\Aspera Connect\bin"

Aspera Connect directory

Take note of your directory.

In order to see all these directories, you may need to "view hidden
files and folders" in Windows Explorer. If you need to do this, then any good
instructional website found through an Internet search should do.

3. Open up a command-line prompt in Windows. If you don't know how to
do this, try the following (Figure 2):
i) press: <Windows key> + R
ii) enter the text: cmd

Windows Command Line prompt


4. In your command line prompt, navigate to that directory which your Aspera program was in.
e.g. in the above example it was:
C:\Users\tobias\AppData\Local\Programs\Aspera\Aspera Connect\bin

I would use the command (Figure 3):
cd "C:\Users\tobias\AppData\Local\Programs\Aspera\Aspera Connect\bin"

The speech marks "" are normally required because there is a directory with a space, "Aspera Connect", and Windows will not understand the directory path otherwise.

Change directory in Command Line to Aspera Connect

5. Now from here, we can enter the Aspera upload command, which is along the
lines of:
ascp -QT -l500m --file-manifest=text -k 2 -o Overwrite=diff <path-to-folder-to-be-uploaded> pride-drop-006@ah01.ebi.ac.uk:<name-of-destination-dir-specified-by-you>

For a new Aspera submission we would have to make a new directory separately on our end, which we would tell you in an email.
Let us pretend that our destination folder is: tobias_batch1

And my dataset I want to upload is in the, "C:\tobias\dataset_to_upload" folder on my PC.

I would then run the command (Figure 4):
ascp -QT -l500m --file-manifest=text -k 2 -o Overwrite=diff C:\tobias\dataset_to_upload pride-drop-006@ah01.ebi.ac.uk:tobias_batch1

Aspera will then prompt you for a password, which we would again need to set up separately in an email.

Your actual command should be something similar to this example (same destination folder, same password, but different folder to be uploaded).

Aspera upload command

The "'--file-manifest=text -k 2" part will generate an aspera progress file on your side that will contain a report on the files that have been uploaded, also you can interrupt the process and then it will only upload the ones that were not there so no more overwriting files. It will also skip the ones that are already in the target directory.

If -l500m ~ 500 Mb/s is unstable and leads to timeouts then I suggest to go back to -l250m as the maximum transfer rate, even that is fast enough to transfer theoretically 2 TBs within a day.

When Aspera finishes, then everything should have been sent. Email pride-support@ebi.ac.uk to inform us that your upload has been successfully finished.

Detailed Linux instructions

This is a detailed example on how to upload your data with Aspera in Ubuntu with FireFox.

1) Install the Aspera Connect tool.
In FireFox, download the Aspera Connect zip archive:
http://downloads.asperasoft.com/connect2/
Extract the zip to a directory, and run the script, e.g.:

    $ ./aspera-connect-3.3.3.81344-linux-64.sh

    Installing Aspera Connect

    Deploying Aspera Connect (/home/tobias/.aspera/connect) for the current user only.
    Restart firefox manually to load the Aspera Connect plug-in

    Install complete.

Then close your FireFox, and relaunch it. Remember the directory where it's installed, which should be in your user's home directory, e.g.: /home/tobias/.aspera/connect/bin

2) Run your Aspera command.
Go to the bin directory where Aspera Connect was installed to, e.g.:

$ cd /home/tobias/.aspera/connect/bin/

Run the command of the format:
./ascp -QT -l500m --file-manifest=text -k 2 -o Overwrite=diff <path-to-folder-to-be-uploaded> pride-drop-006@ah01.ebi.ac.uk:<name-of-target-dir-specified-by-PRIDE>

e.g.
Source directory with my files to be sent: /tobias/Documents/dataset-to-upload/
Target directory on destination server: tobias_batch1

./ascp -QT -l500m --file-manifest=text -k 2 -o Overwrite=diff /tobias/Documents/dataset-to-upload/ pride-drop-006@ah01.ebi.ac.uk:tobias_batch1

Enter the password for the server when prompted.