Don't forget the basics

I have a few rules which I try to abide by in my working life, one of which is 'Keep it simple, stupid!'. There is a great deal to be said for cutting out overcomplicated and unnecessary configuration, particularly when it comes to troubleshooting an existing system.
Don't get me wrong, I enjoy the cut and thrust of deep technical development and configuration as much as the next (slightly geeky) guy, but experience has shown that the business will thank you for a well designed, simple yet efficient solution, and that there are diminishing returns but increased risks as complexity increases.
This isn't to say that some services don't need in depth configuration, more that it's important to know when to stop.

Receive-Side Scaling & MPI

I recently spent an interesting few days discovering, diagnosing & resolving an MPI performance issue on a small Gigabit Ethernet interconnect based cluster. The problem first came to light after running the MPI quick check and Throughput diagnostics from within the HPC Cluster Manager diagnostics suite. Results were significantly down on expected values, returning an average of just over 200 microseconds latency and just over 60MB/s throughput. Expected figures are closer to 50 micoseconds and 105MB/s, so something was quite obviously amiss.
After checking that all appropriate firmware and drivers were up to date the issue was still apparent. Time for some good old fashioned detective work, driving the problem in to an ever smaller box until the answer popped out. Following many combinations of driver version and network setting configuration, the culprit turned out to be the Receive-Side Scaling feature when enabled on newer versions of the driver in question. When turned on and configured to use <1 queue performance was degraded. When turned off, or turned on but configured to use a single queue, performance was as expected. Interestingly when using older versions of the driver RSS could be on and configured to use multiple queues without any performance degradation.

During the investigation I spoke to Xavier Pillons, a Windows Server performance guru at Microsoft, and he came up with some very useful tips which I'm sure he won't mind me sharing:

1.Check the driver release version.
2. Check the TCP Global parameters (use the command line Netsh int tcp show global).
3. On Windows server 2008 try to disable RSS, and play with Chimney Off/On.
4. For a better latency you can disable Interrupt Moderation Rate on the Network Interface.

Problem solved, or at least a workaround found. Happy Days! :)

Diagnostics in HPC Server 2008 R2

A very interesting  post on the Windows HPC Team blog regarding diagnostics in HPC Server V3. I'm really looking forward to third party custom diagnostic tests, should be a very powerful feature indeed. Much of our work is based on management and control of individual jobs and groups of linked jobs, so I'm sure we'll be looking to create appropriate custom tests based on this framework.
I can certainly imagine a scenario where a Systems Center Operations Manager management pack kicks off in house custom tests, then alerts warnings and errors back to operations staff via the SCOM console where appropriate.