Comments on: Distributed exome analysis pipeline with CloudBioLinux and CloudMan http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/ Tue, 04 Jun 2013 12:18:00 +0000 hourly 1 http://wordpress.com/ By: Makefiles to Control Bioinformatic Pipelines | Monkeyologist http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-711 Thu, 29 Nov 2012 05:34:54 +0000 http://bcbio.wordpress.com/?p=217#comment-711 [...] claim as my own work. After a few hours’ research, some early contenders were Bio-Linux and CloudBioLinux, but I was scared off by their size. I wanted something that most anybody could run with minimal [...]

]]> By: Cloud Biocumputing just for you « Elements of Impact http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-522 Fri, 03 Feb 2012 18:59:45 +0000 http://bcbio.wordpress.com/?p=217#comment-522 [...] blue collar bioinformatics’ where he describes the progress being made in online accessible cloud instances (private to you) of analytical environments for bioinformatics. Look for BioCloud Central where [...]

]]> By: Brad Chapman http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-502 Sun, 04 Dec 2011 19:33:47 +0000 http://bcbio.wordpress.com/?p=217#comment-502 Zayed;
That’s exactly right. CloudMan configures a standard SGE cluster with an LSF shared filesystem, and then gives you a web interface to add and remove nodes as needed. If you ssh into the instance, you can run jobs with qsub as you normally would on any other cluster.

The analysis script for this pipeline interacts with SGE in exactly this way. It starts worker jobs with qsub and then sends out messages for processing to them with RabbitMQ.

]]> By: zayedi http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-500 Sat, 03 Dec 2011 05:45:43 +0000 http://bcbio.wordpress.com/?p=217#comment-500 Hi Brad

Very nice tutorial. So Cloudman is the system that brings the cluster on Amazon to life. RabbitMQ is a python library to send messages across to cluster nodes to do their work for alignment, etc.
I’m quite used to the whole qsub paradigm and I assume that if we went up to the part before “Run Analysis” with Cloudman, it would then be a case of just qsubbing shell scripts to the SGE cluster.
Is that correct?

]]> By: Yannick Pouliot http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-474 Tue, 13 Sep 2011 02:21:49 +0000 http://bcbio.wordpress.com/?p=217#comment-474 Will do!

]]> By: Brad Chapman http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-473 Tue, 13 Sep 2011 00:47:57 +0000 http://bcbio.wordpress.com/?p=217#comment-473 Yannick;
I’d be very interested to hear what you come up with if you dig into the image differences more. I haven’t done any benchmarking myself, so it would be cool to see.

]]> By: Yannick Pouliot http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-472 Mon, 12 Sep 2011 17:00:14 +0000 http://bcbio.wordpress.com/?p=217#comment-472 Thx Brad! I’m using a High-Memory Extra Large instance, and the phenomenon is reproducible. Any idea what instance types are best for short-read analysis? Would be an interesting survey to run…

Btw, the I/O performance of HMEL instances is described as “moderate”, and I’m starting to think that this is where the issue lies, as the Large instance has “high” performance I/O…

]]> By: Brad Chapman http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-471 Mon, 12 Sep 2011 12:18:30 +0000 http://bcbio.wordpress.com/?p=217#comment-471 Yannick;
micro instances can often be less than responsive if you have anything substantial running, but they are not meant to be used for processing. With standard large instances I haven’t had any noticeable issues, but I’ve not done a ton of monitoring or metrics. Quora has a nice discussion about this which might be helpful:

http://www.quora.com/Amazon-Web-Services/What-are-some-tips-for-tuning-EC2-EBS-IO-performance

]]> By: Yannick Pouliot http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-469 Mon, 12 Sep 2011 00:27:44 +0000 http://bcbio.wordpress.com/?p=217#comment-469 Great post (really). I have a Q regarding your experience running this on Amazon: do you experience temporary loss of connectivity when running intensive processes? Possibly specific to processes that involve a lot of I/O? I certainly do, though with a my own analytical set-up, and am wondering how widespread this may be…

]]> By: Open Science » Report 09/09/2011 http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-cloudman/#comment-467 Fri, 09 Sep 2011 16:05:37 +0000 http://bcbio.wordpress.com/?p=217#comment-467 [...] http://bcbio.wordpress.com/2011/08/19/distributed-exome-analysis-pipeline-with-cloudbiolinux-and-clo… Posted on 9 de setembro de 2011 by admin. [...]

]]>