With the continued growth of data at EMBL-EBI, the Systems Infrastructure team within the Technical Services Cluster has been looking to alleviate the capacity pressures on some of our increasingly overloaded storage systems.  Due to the number of users and the frequency of use, it is critical that any such changes cause minimal impact to end users, or are, ideally, entirely invisible.  The hunt for a solution has led to the adoption of InfiniteIO, a tool for moving rarely accessed and inactive “cold” data into object storage, thus freeing up primary filesystem space for “hot” (frequently accessed) data.

InfiniteIO scans the primary storage and stores all the filesystem’s metadata in an in-memory database (as well as in the object store for data recovery purposes), subsequently intercepting and responding to metadata operations much faster than the primary storage system itself.

Crucially, all of this happens transparently to the client; the filesystem’s directories appear the same regardless of the storage location of the actual files, with only minor increased latency when pulling data back from the object store.  This means any applications, storage systems or data workflows can continue to use the data as though it is all stored on the primary filesystem.

Testing of a clustered InfiniteIO solution at EMBL-EBI has been taking place for over a year now and preparations to move it into production are currently underway.