Enabling Concurrent Multithreaded MPI Communication on Multicore Petascale Systems

With the ever-increasing numbers of cores per node on HPC systems, applications are increasingly using threads to exploit the shared memory within a node, combined with MPI across nodes. Achieving high performance when a large number of concurrent threads make MPI calls is a challenging task for an MPI implementation. The authors describe the design and implementation of their solution in MPICH2 to achieve high-performance multithreaded communication on the IBM Blue Gene/P. They use a combination of a multichannel-enabled network interface, fine-grained locks, lock-free atomic operations, and specially designed queues to provide a high degree of concurrent access while still maintaining MPI's message-ordering semantics.