Technology:Don Becker On The State Of HPC | Linux Magazine

• Don Becker On The State Of HPC | Linux Magazine

• A Beowulf pioneer provides insights and experience from the HPC trenches.
• Linux magazine HPC Editor Douglas Eadline had a chance recently to discuss the current state of HPC clusters with Beowulf pioneer Don Becker
• DB: Everyone is calling everything they have a cloud strategy. From renting computing time (time-sharing) to managing virtual machines and up-time with completely transparent assist.
• What we have seen, if you have lightweight compute node, is the very first job, if it is a big memory job, like a large matrix computation, we can get a 40-50% performance improvement because we don’t have a dirty virtual to physical mapping. We have a clean set of pages in the virtual to physical mapping because we have done essentially nothing at boot time. That is only one of the ways to accomplish performance, but even with our system that advantage is there only for that very first job.
• As soon as the kernel starts putting objects in memory, they are not completely immovable, but as soon as the kernel grabs a page of memory for itself, you can’t shuffle pages around that allocation. So you get that bump only on the first run.
• With virtualization you have just made this problem one level more difficult.
• Now you have reconcile that with the virtualization people who claim there is no performance impact.
• In HPC, there is a large performance impact because of page table entry thrashing (TLB look aside buffer and caches)
• That is big critical issue and relatively new issue in HPC, it was important before, but now that we are seeing regular deploys of 32 to 128GB of memory per physical machine. With the standard 4K page size that might be 32 million pages you have to manage. If you are stepping through memory that is 32 million mappings that might be pushed in an out of the cache.
  
• Everyone is calling everything they have a cloud strategy.
• If you want guaranteed performance you are renting a machine not a cloud.
  
• It is a big step forward from grid though. The computing community version of grid was we have libraries to make all these different installs communicate rather than doing machine virtualization. Think of it as library level virtualization. For every service they could think of they provide a library function. For every service they did not think of they spent years writing library functions making different and potentially disparate operating systems, distributions, and versions interoperate. I think that turned out to be, in my opinion, a huge failure.
• They could not guarantee consistency, that is they could not guarantee you could run any executable anywhere, and never guaranteed that by running the same executable you would get the same results.
• That is one of the fundamental assumptions you have to make in HPC. If I run a program over there, I have to know what executable is running, what libraries it is linking to and in what order. I need to reproduce that exact same result everywhere in my run.
• Cloud computing provides virtualization at the machine level, you to do more work and it is more of a synchronization rather than a guaranteed consistency but it is a step better than what grids were.
  
• There are both large pages (2 or 4 MByte pages) and giant pages (4 GByte pages). That is an exciting area. I think 4MB pages are sufficient for right now.
• But if we could do 4GB pages, just a handful of Gigabyte pages would solve the problem for large memory jobs and provide predictable execution time and minimal traffic to memory, that is traffic to help manage memory rather than user code memory.
  
• Another area is where everything is going to change for HPC I/O is flash disk (Solid Sate Disk or SSD). Right now they are abysmally bad, Intel has recently updated the firmware on their SSD to solve some of the worst anomalous conditions. They will get better.
  
• But, it changes the I/O expectations from being this very slowing growing curve, we went from 50 MB/sec to 70 MB/sec to 90 MB/s sustained write rate over a period of 8-10 years. We did not even get a factor of 2 sustained write rate on the best drives. So now we are going to see a semiconductor curve instead of the disk drive curve for write rates. That will change everything.
• But it will change with how we have to deal with these rates in the file system structure because the previous models do not apply. We can’t throw it all away because it took several decades to get file systems right, but some dramatic changes are needed.