c - Influence of send modes to MPI overhead in non-blocking point-to-point communication

Question

I want to determine the overhead of non-blocking point-to-point communications in MPI. There are some benchmarks available (like Sandia MPI Micro-Benchmark Suite or OSU micro-benchmarks), but for some reason they do not discriminate between the send modes MPI offers (standard, ready, buffered, synchronous) and only using the standard mode. The MPI report states that

In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been posted, and the data has been moved to the receiver.

I would assume that writing the message into a buffer may have a different performance than sending the message directly to the receiver (which could be physically far away, be connected via a low bandwidth connection, etc). So my question is, whether my assumptions are wrong and there are never any significant performance differences between a buffered send and a ready send (and if yes, why) - or whether these benchmarks just ignore these possible differences (and if yes, why).

Answer 1

You understanding is correct that performance of buffered and unbufferred sends can be different. If you call MPI_Send, the MPI implementation can choose which mechanism to use based on the message size, system architecture, network speed, and many other variables. In general, buffered sends are used for small messages and synchronous sends are used for large messages.

It is recommended that MPI application developers stick with MPI_Send and rely on the MPI implementation to provide the best performance. Consequently, most MPI benchmarks also report the performance of MPI_Send and not the other modes, which are not as commonly used.