binary - packing and unpacking variable length array/string using the struct module in python

Question

I am trying to get a grip around the packing and unpacking of binary data in Python 3. Its actually not that hard to understand, except one problem:

what if I have a variable length textstring and want to pack and unpack this in the most elegant manner?

As far as I can tell from the manual I can only unpack fixed size strings directly? In that case, are there any elegant way of getting around this limitation without padding lots and lots of unnecessary zeroes?

Answer 1

The struct module does only support fixed-length structures. For variable-length strings, your options are either:

• Dynamically construct your format string (a str will have to be converted to a bytes before passing it to pack()):

  s = bytes(s, 'utf-8')    # Or other appropriate encoding
  struct.pack("I%ds" % (len(s),), len(s), s)
  

• Skip struct and just use normal string methods to add the string to your pack()-ed output: struct.pack("I", len(s)) + s

For unpacking, you just have to unpack a bit at a time:

(i,), data = struct.unpack("I", data[:4]), data[4:]
s, data = data[:i], data[i:]

If you're doing a lot of this, you can always add a helper function which uses calcsize to do the string slicing:

def unpack_helper(fmt, data):
    size = struct.calcsize(fmt)
    return struct.unpack(fmt, data[:size]), data[size:]