{"id":199,"date":"2019-10-30T09:16:58","date_gmt":"2019-10-30T09:16:58","guid":{"rendered":"http:\/\/www.ebi.ac.uk\/about\/teams\/its\/?p=199"},"modified":"2021-09-02T08:57:10","modified_gmt":"2021-09-02T08:57:10","slug":"solving-cold-data-storage-problems","status":"publish","type":"post","link":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/blog\/2019\/10\/solving-cold-data-storage-problems\/","title":{"rendered":"Solving cold data storage problems"},"content":{"rendered":"\n<p>With the continued growth of data at EMBL-EBI, the Systems Infrastructure team within the Technical Services Cluster has been looking to alleviate the capacity pressures on some of our increasingly overloaded storage systems.\u00a0 Due to the number of users and the frequency of use, it is critical that any such changes cause minimal impact to end users, or are, ideally, entirely invisible.\u00a0 The hunt for a solution has led to the adoption of <a href=\"https:\/\/infinite.io\/\">InfiniteIO<\/a>, a tool for moving rarely accessed and inactive \u201ccold\u201d data into object storage, thus freeing up primary filesystem space for \u201chot\u201d (frequently accessed) data. <\/p>\n\n\n\n<figure class=\"vf-figure wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"650\" height=\"465\" class=\"vf-figure__image\" src=\"https:\/\/www.ebi.ac.uk\/about\/clusters\/technical-services\/wp-content\/uploads\/2019\/11\/infinteIO.png\" alt=\"\" class=\"wp-image-200\" srcset=\"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-content\/uploads\/2019\/11\/infinteIO.png 650w, https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-content\/uploads\/2019\/11\/infinteIO-300x215.png 300w\" sizes=\"auto, (max-width: 650px) 100vw, 650px\" \/><\/figure>\n\n\n\n<p>InfiniteIO scans the primary storage and stores all the filesystem\u2019s metadata in an in-memory database (as well as in the object store for data recovery purposes), subsequently intercepting and responding to metadata operations much faster than the primary storage system itself.<\/p>\n\n\n\n<p>Crucially, all of this happens transparently to the client; the filesystem\u2019s directories appear the same regardless of the storage location of the actual files, with only minor increased latency when pulling data back from the object store.&nbsp; This means any applications, storage systems or data workflows can continue to use the data as though it is all stored on the primary filesystem.<\/p>\n\n\n\n<p>Testing of a clustered InfiniteIO solution at EMBL-EBI has been taking place for over a year now and preparations to move it into production are currently underway.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>With the continued growth of data at EMBL-EBI, the Systems Infrastructure team within the Technical Services Cluster has been looking to alleviate the capacity pressures on some of our increasingly overloaded storage systems.\u00a0 Due to the number of users and the frequency of use, it is critical&hellip;<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[372],"tags":[2050,373,2049],"embl_taxonomy":[],"class_list":["post-199","post","type-post","status-publish","format-standard","hentry","category-data-storage","tag-cold-data","tag-infinite-io","tag-storage"],"acf":[],"embl_taxonomy_terms":[],"featured_image_src":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-includes\/images\/media\/default.svg","_links":{"self":[{"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/posts\/199","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/comments?post=199"}],"version-history":[{"count":2,"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/posts\/199\/revisions"}],"predecessor-version":[{"id":325,"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/posts\/199\/revisions\/325"}],"wp:attachment":[{"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/media?parent=199"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/categories?post=199"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/tags?post=199"},{"taxonomy":"embl_taxonomy","embeddable":true,"href":"https:\/\/www.ebi.ac.uk\/about\/teams\/its\/wp-json\/wp\/v2\/embl_taxonomy?post=199"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}