Further learning

Events

Communities of practice

Recruiting, developing and training

Competency frameworks

Many of these are now navigable at the EMBL-EBI Competency Hub.

Developing and training

Quality assurance for databases and software

Project management and time-tracking tools

Models/concepts

  • Gannt Charts
  • Agile process – initially used for software development but readily adaptable to other projects. Basis is two-week sprints to get defined chunks of work done.
  • Kanban – better for general operations than projects; process for shifting as many tasks from ‘doing’ to ‘done’ as efficiently as possible
  • Risk register – tool for managing and mitigating risk in a projec

Tools

  • Jira agile project management software – much loved by software developers; free licenses for academics; integration with other Atlassian tools (e.g. confluence wiki); beware – there are lots of third-party plugins that do really cool things but they are expensive!
  • Asana project management app – user friendly; good integration with visualisation tools such as instagannt and diary crosslinking; free licensing only good for teams of 15 or fewer
  • Excel – includes simple templates for, e.g. Gannt charts
  • Github/https://about.gitlab.com/ – single application for the development of the entire DevOps cycle.
  • https://toggl.com/ – time tracking software; can be integrated with numerous other tools
  • https://clickup.com/ – user-friendly alternative to Jira; built with generic project management in mind but includes lots of tools for managing software development projects.
  • Google templates for project management
  • Redmine
  • Trello – simple, visual kanban-style board; not ideal for complex projects but OK if you just have to get through tons of tasks or tickets.
  • Bugherd – neat combination of project management and collaborative bug tracking; great for development of web-based services
  • Basecamp – project management and team communication; central storage of shared documents

Compute

  • Docker cannot be run on HPC unless you give your users root access (which you don’t want to do!); Singularity was designed to enable containerised processes on HPC. Docker is moving in this direction but is not there yet.
  • Resourcing the use of commercial cloud – you need to optimise use of commercial cloud services to ask for the right resources for the job; you need to calculate how your user is billed for the job that they’re doing (or that you’re doing for them). Platform as a service enables you to ‘get stuff done’ whilst retaining a lot of flexibility; containers as a service are a subtype of platform as a service. Kubernetes enables you to manage containers as a service.
  • You also need a workflow language to string containerised services together, e.g. common workflow language. Better to push the task to the data than to pull the data to the task.
  • Software as a service – EMBL-EBI tools as a classic example; user operates everything over the web.
  • Hybrid cloud – can you set up an entire workflow and port it from your own compute to the public cloud if you need more compute than your system can provide? But how do you manage large data sets in hybrid-cloud mode? There are trade-offs to be made; better to use AWL’s standard protocol than NFS.
  • If you know where your data are, or if all your data are in one place, Galaxy, NextFlow etc. are fit for purpose; if your data are distributed in many places and data discovery becomes part of your workflow, you need something more scalable that allows for the use of in-house or external clusters.
  • NF-core – community effort to collect a curated set of analysis pipelines built using Nextflow
  • Packaging up different versions of software for different types of user – Bioconda –  https://bioconda.github.io/
  • The EMBL-EBI ResOps cloud consultancy team is currently in the process of creating a standalone version of its ResOps that can be watched on-demand, but  it will be some time before this is released.  The material for self-study is at https://bit.ly/resops-2019, and if anyone wants to give it a go Tony Wildish has very kindly offered to help  with any questions: please send them to cloud-consultants@ebi.ac.uk.
  • There are also courses from Coursera, Linux Academy and Kube Academy on getting cloud ready; it’s worth checking the course pre-requisites first.

Papers and other pearls of wisdom

Papers

Blogs

Reports, forums and other resources

The following resources were provided by Nicolas Descostes