python - _pickle in python3 doesn't work for large data saving

Question

I am trying to apply _pickle to save data onto disk. But when calling _pickle.dump, I got an error

OverflowError: cannot serialize a bytes object larger than 4 GiB

Is this a hard limitation to use _pickle? (cPickle for python2)

Answer 1

Not anymore in Python 3.4 which has PEP 3154 and Pickle 4.0
https://www.python.org/dev/peps/pep-3154/

But you need to say you want to use version 4 of the protocol:
https://docs.python.org/3/library/pickle.html

pickle.dump(d, open("file", 'w'), protocol=4)