Know your underlying infrastructure

So, you're looking to deploy and support a Windows HPC Server cluster, but you have a sneaky suspicion that there's more under the covers that you bargained for. What do you know, your instincts are correct, and you're suddenly in a world of Microsoft technologies which you should at least be aware of. 
The good news is (and this is a big advantage) that these underlying technologies are common, and it may be that your company/organisation has experience in those areas, for example in a corporate IT team. If those skills can be leveraged for your Windows HPC deployment, then it's all gravy - you can stick to what you do best & eke out tip top performance, and write some killer submission scrips for your cluster users.
But wait, what if you have no such skills in house? Well, here's a quick rundown of some of the things involved...


Windows Server 2008
The base Operating System. Note that Windows HPC Server can run on various editions of the OS (e.g. HPC Edition, Standard Edition, Enterprise Edition), which should show that the HPC Pack is a separate entity to the OS. License wise it's important to note that you can only run Windows HPC related technologies on 2008 HPC Edition - no chance of saving a few notes by using it as a base OS for your corporate Exchange system ;)
When thinking about the OS, pay particular attention to driver versions and settings. Use the built in reliability, performance & logging tools to your advantage.


Active Directory
There's no getting round the Active Directory thing. It's at the core of everything Windows HPC Server does, from deployment to running jobs to data authorisation. There are several potential options here depending on your environment. If you have a corporate AD I would strongly suggest that you work with your corp IT guys to integrate the cluster. This type of configuration will smooth the wheels of progress significantly. Allowing an existing authentication regime, one in which users already have accounts set up, can save a bunch of user admin overhead. If this is not an option, it's worth spending at least a bit of time pondering your AD architecture. It's nice and easy to simply promote your headnode to a domain controller, but I would suggest that you also run up another, separate Domain Controller, as losing your AD can be a royal PITA to recover from.
Either way, take some time to get this bit right, and to learn the basics of AD operation & you'll likely see payback in future.


SQL Server
I'm planning to dive into SQL Server a bit deeper in another post, but suffice to say it's well worth picking up some knowledge in this area.


Windows Deployment Services
WDS provides the platform for the super slick node deployment mechanism within Windows HPC Server. It's wrapped so nicely that you may never need to poke about under the covers, but definitely pay attention to imagex, diskpart, the \\<headnode>\reminst share and it's contents.


DHCP and DNS
OK so these are not necessarily Windows specific, but getting your DNS and DHCP knowledge down is very useful. I'm going to post about network configuration in another post, so will try to include some DNS / DHCP tips there too.


RRAS
Routing and Remote Access Services is an umbrella for bunch of useful Windows features. These include Dialup; VPN (both client server and site to site); IP subnet routing, and Network Address Translation (NAT). In the case of Windows HPC Server the NAT part is of interest. It plays an integral part in operation of those network topologies which rely on compute nodes only having connection to the private network. In these cases traffic destined for hosts on subnets other then the private network travel via the headnode (as gateway), and NAT out onto the enterprise network. This may be pertinent e.g. for Windows Updates and the like.


Windows Failover Clustering
If you're going for a High Availability head node solution, take some time to become acquainted with how Windows Failover Clustering works. Behaviour and types of shared disks; should you used disk share, quorum, node majority; Failover Cluster network configuration; Cluster resource DNS registration; verification and support of cluster components. This is a big subject in itself, and an awareness of how the technology works is important.

It's all in the name

I like names, they have a funny way of telling a story in a single word. Take my name for instance, I'm sure you read the word 'Dan' and assume that I'm a strong, intelligent, handsome guy who's no end of fun to hang out with, right? Or maybe it's the other way round, and the person defines your perception of a name. I mean, if all people named Dan are super cool, does that make the name Dan super cool?

Anyway, what I really want to talk about here is how Windows HPC handles name resolution, particularly across private and application networks.

First off let's think a little about general Windows host name resolution order. This looks like this, listed in order:

1. Checks it's own name.
2. Looks in the Local DNS cache (you can list entries using ipconfig /displaydns).
3. Local HOSTS file (C:\Windows\System32\Drivers\etc\Hosts)
4. Adds the Search Suffix configured on the machine (if not FQDN), and query DNS
5. WINS (NetBios name resolution)
6. Broadcast on local subnet
7. Local LMHOSTS file (C:\Windows\System32\Drivers\etc\Hosts)

Pretty thorough I'm sure you'll agree.

Now let's look at this with our high performance and management hats on. We want to be able to get an  answer to our name resolution queries as quickly as possible, while ensuring consistency across all nodes in the cluster. We also need to resolve a host name to the appropriate network address for the cluster network we're after. Running through the resolution order above we can ignore 1. for obvious reasons. 2. may be interesting performance wise,  but as the cache may not contain records for all nodes, it's an inconsistent choice. 3. hmmm, the good old local hosts file, sounds kinda antiquated and simplistic don't you think? But it's at number 3 on the list, crucially checked before DNS resolution. And maybe it can be managed by one of the HPC services running on all nodes? Oh, this is starting to sound decent. Just to be sure though let's continue. 4. is our old friend DNS, which sounds like the way to go. But each DNS lookup can take a relatively long time. Seems like it'd be good for management, but as good as a cluster managed solution? Once we get to 5, 6 and 7 things are drifting off into desperation, so let's not say too much about those guys.

Well what do you know, Hosts seems to be a very good choice here, and lo and behold that's how it works in practice! Check out this example hosts file taken from a handy dev cluster... 

# Copyright (c) 1993-1999 Microsoft Corp.

# 

# This host file is maintained by the Compute Cluster Configuration

# Management Service. Changes made to the file that match the netbios names

# for existing nodes in the cluster will be removed and replaced by entries

# calculated by the management service.

# 

# Modify the following line to set the property to 'false' to disable this

# behavior. This will prevent the management service from making any

# further modifications to the file

# 

# ManageFile = true

127.0.0.1                localhost

192.168.5.23             HPCDEV-HN02                    #HPC

192.168.100.11           HPCDEV-HN02                    #HPC

192.168.0.11             HPCDEV-CN001                   #HPC

192.168.0.10             HPCDEV-CN002                   #HPC

192.168.0.134            HPCDEV-CN003                   #HPC

192.168.0.1              HPCDEV-HN01                    #HPC

192.168.0.2              HPCDEV-HN02                    #HPC

192.168.0.3              HPCDEV-VHN01                   #HPC

192.168.1.1              HPCDEV-HN01                    #HPC

192.168.1.2              HPCDEV-HN02                    #HPC

192.168.1.3              HPCDEV-VHN01                   #HPC

192.168.5.22             Enterprise.HPCDEV-HN01         #HPC
192.168.5.23             Enterprise.HPCDEV-HN02         #HPC

192.168.5.28             Enterprise.HPCDEV-VHN01        #HPC

192.168.0.11             Private.HPCDEV-CN001           #HPC

192.168.0.10             Private.HPCDEV-CN002           #HPC

192.168.0.134            Private.HPCDEV-CN003           #HPC

192.168.0.1              Private.HPCDEV-HN01            #HPC

192.168.0.2              Private.HPCDEV-HN02            #HPC

192.168.0.3              Private.HPCDEV-VHN01           #HPC

192.168.1.11             Application.HPCDEV-CN001       #HPC

192.168.1.10             Application.HPCDEV-CN002       #HPC

192.168.1.134            Application.HPCDEV-CN003       #HPC

192.168.1.1              Application.HPCDEV-HN01        #HPC

192.168.1.2              Application.HPCDEV-HN02        #HPC

192.168.1.3              Application.HPCDEV-VHN01       #HPC

This file reflects the current addressing of a HA head node cluster configuration which has three compute nodes. Network topology is 3. Compute nodes isolated on private and application networks. Interesting to note that only the active head node addresses are detailed in the standard format listing (HPCDEV-HN02) for networks other than private and application (in this case Failover cluster heartbeat network and enterprise).

Check out those funky Enterprise. Private. and Application. entries. This allows the cluster service to be very specific in its address resolution requests, assuring it will always get back the address on an appropriate network.

But what if you want to host non HPC Server managed machines on your private network, therefore requiring hosts to register in DNS (they do not by default)? Well, you can use the awesomeness that is powershell...

Set-HpcNetwork -PrivateDnsRegistrationType WithConnectionDnsSuffix

One thing to beware of - check out the warning at the top of the hosts file. If you set  Managefile = False and manually alter entries previously managed by HPC Server things may get a little broken.