CRAM Reference Registry reverse proxy

To reduce network traffic originating from the use of the CRAM Reference Registry we recommend using locally cached reference sequences. In addition to local caches supported by the SAMTools and CRAMToolkit it is possible to cache sequences using an HTTP proxy.

Instructions are provided below for setting up a local cache using Squid HTTP proxy on Linux. The Squid is used as a reverse proxy to cache reference sequences retrieved from the CRAM Reference Registry.

Setting up Squid reverse proxy

1. Download and install Squid

If a binary package is not available for download then Squid can also be compiled from source.  We recommend that you install version 3.1 or later.

2. Create directories for the cache files, log files and the Squid configuration file

In the example below we have created directories under /data/squid but any directory accessible to the user running Squid can be used after it has been defined in the Squid configuration file.

Directory Description
/data/squid/cache Used to store the local squid cache
/data/squid/log Used to store the Squid log files
/data/squid/conf Used to store the Squid configuration file

3. Create the Squid configuration file

The following configuration file in /data/squid/conf will cache locally reference sequences retrieved from the CRAM Reference Registry. Parameter values that should be configured locally are bolded.

max_filedesc 4096
cache_mem <memory>
maximum_object_size_in_memory 512 MB
maximum_object_size 1024 MB
http_port <port> accel defaultsite=<hostname> vhost
#forwarded_for on
cache_peer www.ebi.ac.uk parent 80 0 no-query originserver name=myAccel
cache_dir ufs /data/squid/cache 65536 16 256
acl mysites dstdomain ebi.ac.uk
acl all src all
http_access allow mysites
http_access allow all
cache_peer_access myAccel allow mysites
cache_peer_access myAccel allow all
access_log /data/squid/log/access.log
cache_store_log /data/squid/log/store.log
cache_log /data/squid/log/cache.log
Parameter Example  

<port>

8080

The HTTP port the Squid server is configured to use. Please note that ports below 1024 require root privileges.

<hostname>

 

Must match the hostname of the machine running the Squid server.
<memory> 10048 MB The amount of memory reserved for Squid cache. It is recommended that this does not exceed 70% of the available physical memory. If the number of objects that is being cached exceeds the available memory then the cache will write the retrieved reference sequences to the Squid cache directory.  

4. Initialize Squid

Execute the following command:

squid -f /data/squid/conf/squid.conf -z

5. Start up Squid

Execute the following command:

squid -f /data/squid/conf/squid.conf

This will start the squid process as a daemon so that it runs in the background. If you wish you can execute squid in the foreground by adding the parameter -N. 

6. Configuring SAMTools to use Squid

In SAMTools, the REF_CACHE environment variable is used to indicate that any downloaded reference sequences are stored locally in the specified directory in order to avoid subsequent downloads.

If you wish SAMTools to cache retrieved sequences in addition to caching them in Squid then you should define both REF_PATH and REF_CACHE environment variables:

export REF_PATH=<dir>/%2s/%2s/%s:http://<hostname>:<port>/ena/cram/md5/%s
export REF_CACHE=/<dir>/%2s/%2s/%s

where <hostname> and <port> are the ones used by the Squid server, and the <dir> is the directory path used by SAMTools for its cache. The %2s/%2s/%s syntax informs SAMTools to store the sequences in sub-directories first using the first two digits and then using the second two digits of the MD5 checksum. This helps to avoid one large directory with thousands of files in it.

If you wish to cache reference sequences only in the Squid cache then only the REF_PATH environment variable should be defined:

export REF_PATH=http://<hostname>:<port>/ena/cram/md5/%s

7. Configuring CRAMToolkit to use Squid

CRAMTools can be instructed to use the Squid cache by defining the following Java propertly:

java -DREF_URL_TEMPLATE=http://<hostname>:<port>/ena/cram/md5/%s

where <hostname> and <port> are the ones used by the Squid server.

8. Testing Squid

Execute the following command to test the cache:

curl -v http://<hostname>:<port>/ena/cram/md5/9aff30af7b14df874594c8ed07b279ba | md5

The computed md5 checksum should match the requested entry.  The cache status of the request can be determined by checking the /data/squid/log/access.log file. The cache status is available in the fourth column of the access log and can be one of the following:

Status Description
TCP_MISS/200 The requested object is not available in the cache, the request was forwarded on to the cache peer.
TCP_HIT/200 The requested object is available in the Squid disk cache.
TCP_MEM_HIT/200 The requested object is available in the Squid memory cache.

9. Shut down Squid

To gracefully shut down the squid daemon process, execute the following command:

squid -k shutdown

Latest ENA news

11 Oct 2017: Read data download issues resolved

Read data download issues previously affecting ftp.sra.ebi.ac.uk and fasp.sra.ebi.ac.uk services now resolved.

06 Oct 2017: ENA read data download issues

Issues with read data download from ftp.sra.ebi.ac.uk and fasp.sra.ebi.ac.uk

04 Oct 2017: ENA Release 133

Release 133 of ENA's assembled/annotated sequences now available