c - Passing variable-length structures between MPI processes

Question

I need to MPI_Gatherv() a number of int/string pairs. Let's say each pair looks like this:

struct Pair {
  int x;
  unsigned s_len;
  char s[1]; // variable-length string of s_len chars
};

How to define an appropriate MPI datatype for Pair?

Answer 1

In short, it's theoretically impossible to send one message of variable size and receive it into a buffer of the perfect size. You'll either have to send a first message with the sizes of each string and then a second message with the strings themselves, or encode that metainfo into the payload and use a static receiving buffer.

If you must send only one message, then I'd forgo defining a datatype for Pair: instead, I'd create a datatype for the entire payload and dump all the data into one contiguous, untyped package. Then at the receiving end you could iterate over it, allocating the exact amount of space necessary for each string and filling it up. Let me whip up an ASCII diagram to illustrate. This would be your payload:

|..x1..|..s_len1..|....string1....|..x2..|..s_len2..|.string2.|..x3..|..s_len3..|.......string3.......|...

You send the whole thing as one unit (e.g. an array of MPI_BYTE), then the receiver would unpack it something like this:

while (buffer is not empty)
{
    read x;
    read s_len;
    allocate s_len characters;
    move s_len characters from buffer to allocated space;
}

Note however that this solution only works if the data representation of integers and chars is the same on the sending and receiving systems.