Sunday, July 26, 2009

Unbound Development Environment for HPC

Software development can really be a fun if you get your development environment available to you anywhere/everywhere. This is specially true for a typical HPC Developer who not only has to worry about right compilers, build environment, test environment, dependencies etc. but also the complex logic of parallel applications.

So many times you will find that you are tied to your development environment because creating a new one could be just a hassle. More ever, the time required to create such development environment may not be justified if you are running on tight schedule. The same goes with the test environment. I mean, there is no way of testing your application quickly on the parallel environment without actually submitting the job to a real cluster. I know you can run multiple threads of MPI on same system, but that's not a real good test. What if you have an application which reads/writes to a local file system (say, in /opt) in which case you actually need multiple hosts to test if it does that I/O on individual host properly.

Now what if someone bundles compilers and IDE (with advanced D-Trace and SGE plugins) in an virtual machines and ships it? Well then you can setup up your own dependencies and build environment in that Virtual Machine. There you go... now you can download this Virtual Machine on your laptop, desktop or run it on some VMware Infrastructure and access it using your netbook. You can make backup copies of this virtual machine too, and there is one thing you can do if you are working with Virtual Machines which is impossible to do with physical systems.... you can use snapshot to go back in time. So if you change something in your development environment in say, while developing version 2.0 and now say you want settings from version 1.0, all you have to do is restore the snapshot! I am talking about Sun HPC Software, Developer Edition for OpenSolaris 1.0. But that's not all...

Earlier I was talking about test environments. Sun HPC Software, Developer Edition for OpenSolaris 1.0 comes with pre-configured/ready-to-use 2 nodes cluster that is embedded within this Virtual Machine. This mini cluster is for testing your parallel applications right there in your development environment. So that even if you are disconnected from rest of the world you are still a job submit away from your test cluster! This test environment comes with pre-installed and pre-configured installation of OpenMPI and Sun Grid Engine.

There is something really interesting about this test environment... It can help you test your parallel application even if you don't get that revered slot on your organization's HPC cluster. Test environment in that Virtual Machine comes with Sun Grid Engine SGE's Cloud Adapter. This Cloud Adapter helps you to create your own HPC Cluster in the Cloud using Amazon EC2, with just a click of a mouse!

You can download the Sun HPC Software, Developer Edition for OpenSolaris 1.0 from here.

No comments: