I want to save a dict
or arrays.
I try both with np.save
and with pickle
and see that the former always take much less time.
My actual data is much bigger but I just present a small piece here for demonstration purposes:
import numpy as np
#import numpy.array as array
import time
import pickle
b = {0: [np.array([0, 0, 0, 0])], 1: [np.array([1, 0, 0, 0]), np.array([0, 1, 0, 0]), np.array([0, 0, 1, 0]), np.array([0, 0, 0, 1]), np.array([-1, 0, 0, 0]), np.array([ 0, -1, 0, 0]), np.array([ 0, 0, -1, 0]), np.array([ 0, 0, 0, -1])], 2: [np.array([2, 0, 0, 0]), np.array([1, 1, 0, 0]), np.array([1, 0, 1, 0]), np.array([1, 0, 0, 1]), np.array([ 1, -1, 0, 0]), np.array([ 1, 0, -1, 0]), np.array([ 1, 0, 0, -1])], 3: [np.array([1, 0, 0, 0]), np.array([0, 1, 0, 0]), np.array([0, 0, 1, 0]), np.array([0, 0, 0, 1]), np.array([-1, 0, 0, 0]), np.array([ 0, -1, 0, 0]), np.array([ 0, 0, -1, 0]), np.array([ 0, 0, 0, -1])], 4: [np.array([2, 0, 0, 0]), np.array([1, 1, 0, 0]), np.array([1, 0, 1, 0]), np.array([1, 0, 0, 1]), np.array([ 1, -1, 0, 0]), np.array([ 1, 0, -1, 0]), np.array([ 1, 0, 0, -1])], 5: [np.array([0, 0, 0, 0])], 6: [np.array([1, 0, 0, 0]), np.array([0, 1, 0, 0]), np.array([0, 0, 1, 0]), np.array([0, 0, 0, 1]), np.array([-1, 0, 0, 0]), np.array([ 0, -1, 0, 0]), np.array([ 0, 0, -1, 0]), np.array([ 0, 0, 0, -1])], 2: [np.array([2, 0, 0, 0]), np.array([1, 1, 0, 0]), np.array([1, 0, 1, 0]), np.array([1, 0, 0, 1]), np.array([ 1, -1, 0, 0]), np.array([ 1, 0, -1, 0]), np.array([ 1, 0, 0, -1])], 7: [np.array([1, 0, 0, 0]), np.array([0, 1, 0, 0]), np.array([0, 0, 1, 0]), np.array([0, 0, 0, 1]), np.array([-1, 0, 0, 0]), np.array([ 0, -1, 0, 0]), np.array([ 0, 0, -1, 0]), np.array([ 0, 0, 0, -1])], 8: [np.array([2, 0, 0, 0]), np.array([1, 1, 0, 0]), np.array([1, 0, 1, 0]), np.array([1, 0, 0, 1]), np.array([ 1, -1, 0, 0]), np.array([ 1, 0, -1, 0]), np.array([ 1, 0, 0, -1])]}
start_time = time.time()
with open('testpickle', 'wb') as myfile:
pickle.dump(b, myfile)
print("--- Time to save with pickle: %s milliseconds ---" % (1000*time.time() - 1000*start_time))
start_time = time.time()
np.save('numpy', b)
print("--- Time to save with numpy: %s milliseconds ---" % (1000*time.time() - 1000*start_time))
start_time = time.time()
with open('testpickle', 'rb') as myfile:
g1 = pickle.load(myfile)
print("--- Time to load with pickle: %s milliseconds ---" % (1000*time.time() - 1000*start_time))
start_time = time.time()
g2 = np.load('numpy.npy')
print("--- Time to load with numpy: %s milliseconds ---" % (1000*time.time() - 1000*start_time))
which gives an output:
--- Time to save with pickle: 4.0 milliseconds ---
--- Time to save with numpy: 1.0 milliseconds ---
--- Time to load with pickle: 2.0 milliseconds ---
--- Time to load with numpy: 1.0 milliseconds ---
The time difference is even more pronounced with my actual size (~100,000 keys in the dict).
Why does pickle take longer than np.save, both for saving and for loading?
When should I use pickle
?
See Question&Answers more detail:
os