Sun Grid Engine gets fast (and fat!) for 6.0

Sun Grid Engine 6 reminds me of a big American SUV in many ways. It’s big, heavy, and often overkill, but at the same time incredibly practical. And like its real-world cousins, these days SGE is pretty damned fast, too. Just don’t get in front of it; it ain’t gonna stop for nobody:

Job scheduling/queueing is such a convenient mechanism for so many things in scientific and enterprise computing that I’ve always been surprised things like SGE aren’t used more. Or even integrated into the OS. But then I deploy SGE or Torque or LSF or something and I’m outraged by the general crappyness of these things. They always seem to be buggy as all hell, ignore tons of corner cases, and are generally cumbersome and frustrating to learn and deploy. This was certainly the case when I settled on SGE 5 as the “lesser evil” a few years ago for the GSC… We picked it mostly because it crashed less than OpenPBS (Torque/Maui were still very immature) and we at least had the option of converting our “free” install into a paid-for commercial product supported by Sun if we were stuck and the community forums couldn’t help.

But we’ve certainly had issues, and, yes, at times I’ve ranted that SGE must have been written by a bunch of complete UNIX newbies. Not exactly confidence inspiring, but serve us well it did for a few years and overall I couldn’t complain considering the price ($0).

We started looking at alternatives, but then Sun announced their “Sun Compute Utility” initiative, and I thought “Hmmmm.. this could be good, because now Sun themselves will be motivated to fix a lot of the problems in SGE if they’re hoping to scale anywhere near what they’re imagining for this new product!”. So when SGE 6 was released a while back, we starting testing. First in a VMware cluster, then on our partitioned legacy cluster, and then on our 64-bit Opteron nodes.

I’m happy to report that we finished migrating the entire cluster from Sun Grid Engine 5.3 to SGE 6.0u8 last week. So far, things look really good. Commands that used to take 10 or 20 seconds to run now return instantly. Stress testing passed with flying colours with about a hundred thousand jobs “in play” and jobs starting/stopping at about 20 per second. Overall, SGE6 is a big improvement in stability, speed, and functionality. The XML output options from qstat and friends is great, as is the DRMAA support. We already have a DRMAA app written to talk to SGE directly (using the Java API and wrapped in a Perl script for handling signals).

So I feel good about SGE again. But if you’re considering upgrading to version 6 or migrating to SGE from another scheduler, please do make sure your head node has sufficient RAM. As you can see in the screenshot at the top, it’s pretty easy to burn 1.5GB of memory with SGE 6. The good news is, RAM is cheap!

We’re running SGE 6 on Linux, BTW, and the install process is smooth and painless now (unlike SGE 5).

Next steps: drmaa4ruby