A job a day keeps the user at bay

There's always one isn't there? A pesky user who wants to try something fancy on a cluster but isn't entirely sure how to do it. This is the coal face of HPC. It might not have the glamour of procurement, or the technical challenge of system administration, but being able to help a stuck user is possibly the most rewarding part of the job.
I like to set myself a challenge every now and then (job a day? well, maybe not quite that often) to submit a job with crazy parameters, or oddball resource requirements. You never know when a user will want to do something similar, and then a) you'll be able to help, and b) you'll look pretty smart ;)
Thinking more about this I might fire in a blog post every now and than containing an interesting job submission technique, might be interesting to see if anyone can come up with any nifty alternatives...

Know your underlying infrastructure

So, you're looking to deploy and support a Windows HPC Server cluster, but you have a sneaky suspicion that there's more under the covers that you bargained for. What do you know, your instincts are correct, and you're suddenly in a world of Microsoft technologies which you should at least be aware of. 
The good news is (and this is a big advantage) that these underlying technologies are common, and it may be that your company/organisation has experience in those areas, for example in a corporate IT team. If those skills can be leveraged for your Windows HPC deployment, then it's all gravy - you can stick to what you do best & eke out tip top performance, and write some killer submission scrips for your cluster users.
But wait, what if you have no such skills in house? Well, here's a quick rundown of some of the things involved...


Windows Server 2008
The base Operating System. Note that Windows HPC Server can run on various editions of the OS (e.g. HPC Edition, Standard Edition, Enterprise Edition), which should show that the HPC Pack is a separate entity to the OS. License wise it's important to note that you can only run Windows HPC related technologies on 2008 HPC Edition - no chance of saving a few notes by using it as a base OS for your corporate Exchange system ;)
When thinking about the OS, pay particular attention to driver versions and settings. Use the built in reliability, performance & logging tools to your advantage.


Active Directory
There's no getting round the Active Directory thing. It's at the core of everything Windows HPC Server does, from deployment to running jobs to data authorisation. There are several potential options here depending on your environment. If you have a corporate AD I would strongly suggest that you work with your corp IT guys to integrate the cluster. This type of configuration will smooth the wheels of progress significantly. Allowing an existing authentication regime, one in which users already have accounts set up, can save a bunch of user admin overhead. If this is not an option, it's worth spending at least a bit of time pondering your AD architecture. It's nice and easy to simply promote your headnode to a domain controller, but I would suggest that you also run up another, separate Domain Controller, as losing your AD can be a royal PITA to recover from.
Either way, take some time to get this bit right, and to learn the basics of AD operation & you'll likely see payback in future.


SQL Server
I'm planning to dive into SQL Server a bit deeper in another post, but suffice to say it's well worth picking up some knowledge in this area.


Windows Deployment Services
WDS provides the platform for the super slick node deployment mechanism within Windows HPC Server. It's wrapped so nicely that you may never need to poke about under the covers, but definitely pay attention to imagex, diskpart, the \\<headnode>\reminst share and it's contents.


DHCP and DNS
OK so these are not necessarily Windows specific, but getting your DNS and DHCP knowledge down is very useful. I'm going to post about network configuration in another post, so will try to include some DNS / DHCP tips there too.


RRAS
Routing and Remote Access Services is an umbrella for bunch of useful Windows features. These include Dialup; VPN (both client server and site to site); IP subnet routing, and Network Address Translation (NAT). In the case of Windows HPC Server the NAT part is of interest. It plays an integral part in operation of those network topologies which rely on compute nodes only having connection to the private network. In these cases traffic destined for hosts on subnets other then the private network travel via the headnode (as gateway), and NAT out onto the enterprise network. This may be pertinent e.g. for Windows Updates and the like.


Windows Failover Clustering
If you're going for a High Availability head node solution, take some time to become acquainted with how Windows Failover Clustering works. Behaviour and types of shared disks; should you used disk share, quorum, node majority; Failover Cluster network configuration; Cluster resource DNS registration; verification and support of cluster components. This is a big subject in itself, and an awareness of how the technology works is important.

Don't forget the basics

I have a few rules which I try to abide by in my working life, one of which is 'Keep it simple, stupid!'. There is a great deal to be said for cutting out overcomplicated and unnecessary configuration, particularly when it comes to troubleshooting an existing system.
Don't get me wrong, I enjoy the cut and thrust of deep technical development and configuration as much as the next (slightly geeky) guy, but experience has shown that the business will thank you for a well designed, simple yet efficient solution, and that there are diminishing returns but increased risks as complexity increases.
This isn't to say that some services don't need in depth configuration, more that it's important to know when to stop.

Try to get to SuperComputing.

I was fortunate enough to attend the SuperComputing conference in Portland, OR last year, and can honestly say it was a blast. I've been to several similar large events in my time, and SC09 was right up there amongst the best. I'm not sure if it was the location (I loved Portland, what an awesome city!), the people (hung out with some fun individuals), or the conference itself, but it was fantastic all round.
I'll be hoping to get out to New Orleans for SC10, maybe I'll see you there...

Damn there are some clever people out there

The technical computing world is populated with some scarily intelligent people, seemingly brought up from an early age on complex algorithms and deep mathematical theory. Unfortunately I am not, and will never be, one of these people no matter how hard I try, so I have to look for other ways to make myself useful.
Luckily, skills developed in the enterprise world are also useful when managing Windows HPC Server, as a lot of the building blocks are common. Aside from a technical tool belt armed with AD, SQL Server, WDS, DNS etc etc, some of the softer skills can be very useful too. Knowledge and experience of support processes, project management, incident resolution, dealing with customers, and service delivery are all as applicable in the HPC world as any other environment.
After all, your carefully constructed and configured cluster is providing a service to your customers which is often as critical as any enterprise offering.

High Performance is Relative

Whenever you read big announcements and stories about HPC in the press, they tend to concentrate on mega clusters dedicated to solving some of the giant problems of our times. It may be climate modelling, genome research, finding cures for terrible diseases, or unlocking the secrets of the universe. All very worthy causes I'm sure, but chances are your problems are slightly smaller scale, yet still require heroic solutions. Maybe you need to design your new product in half the time, or you need to make your current product 50% more efficient. You might be looking to calculate insurance premium returns, or recover from the global downturn in as short a time as possible.
Whatever you hope to achieve from HPC, the results can be heroic on a personal level, even if it just means you get a well done from the boss!

Trust the product group

This one speaks for itself. My experience of the Windows HPC Product Group has been consistently excellent.  The official Windows HPC team blog is a goldmine of information, and questions to the Windows HPC forums are often investigated & answered by team members. If you have a question or issue, don't hesitate to ask on the forums.
These guys have always been very willing to help wherever possible, and for that they deserve credit.

The Hit Parade

So, I've been thinking about some stuff about my time as a Windows HPC Sysadmin which I'd like to share, & the more I think, the more top level bullet points I come up with. I thought I'd kick off this blog with a couple of top tens. The first are somewhat abstract topics, the second specific to Windows HPC Server. I'll then follow it up with some more detailed blog posts which dive a little deeper into each of them.

So, without further ado, I give to you:
The Ballpark, and The Strike Zone

The Ballpark

Increase your generic High Performance Computing smarts.
Many new Windows HPC Sysadmins will be heading in from the Enterprise, having expertise in Microsoft platforms but relatively little knowledge of HPC.

Trust the product group.
You know, no-one's perfect, but the HPC product group come pretty close.

High Performance is relative.
You don't necessarily need a multi-thousand core set up to produce heroic results.

You get out what you put in.
Contributing to the Windows HPC forum has been a very useful process.

Damn there are some clever people out there.
Sometimes I just feel inadequate, desperately trying to wrap my feeble brain around complex problems discussed by some of my peers.

Do you really need it?
There's a bunch of people out there intent on selling you a bunch of kit, but is it right for you?

The methodical process of troubleshooting.
My advice - start from the bottom & work up... or start at the top and work down.

Don't forget the basics.
There's pleasure to be found in the simple things in life. Or, if you want to be blunt, keep it simple, stupid!

The business is (nearly) always right.
I find there's direct correlation between good business reporting and being left alone to get on with work.

Try to get to SuperComputing.
No, really, it's a great show, with great people.


The Strike Zone

Know your underlying infrastructure.
You only need to look at the Top 500 results to know that alot of HPC kit does not currently run Windows HPC Server. This means that many Windows HPC Sysadmins will be transferring their expertise from Linux (or other OS) based systems.

It's all in the name
How does Windows HPC handle name resolution?

Digging through the versions
SP1? Well, yes, but SP1 of what? And R2? Is that beta or RTM? So you want to install what on where now?

Performance enhancing shrugs
Well, it seems to run OK, but is OK good enough?

The database interloper
SQL Server? In every Windows HPC Server deployment? Surely not.

PowerShell is powerful
OK it might sound like an obvious statement, but it's true! At the risk of offending those not listed I have some favourite HPC PowerShell Commandlets I'd like to share.

Network topology - choices choices
RRAS, dedicated router, managed firewall and the like.

Monitor Lizard, the thinking person's test rig
If you've ever tried to compile HPL to run on Windows HPC you'll absolutely love Lizard. Performance results aside, it's a great starting point for cluster verification.

Node deployment
If one person with one DVD takes 2 hours to build one compute node, how long would it take that person to build a tree house?

A job a day keeps the user at bay
Become one with your users, know what they know, type what they type.