Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
459 views
in Technique[技术] by (71.8m points)

python - reshaping a view of a n-dimensional array without using reshape

tl;dr Can I reshape a view of a numpy array from 5x5x5x3x3x3 to 125x1x1x3x3x3 without using numpy.reshape?

I would like to perform a sliding window operation (with different strides) to a volume (size of MxMxM). The sliding window array can be generated with the use of numpy.lib.stride_tricks.as_strided, as previously suggested by Benjamin and Eickenberg, and demonstrated in the below code snippet, which uses a helper method from skimage that uses as_strided.

The output from this helper method gives me a shape of NxNxNxnxnxn, but I'd prefer the shape to be N^3x1xnxnxn. While I can use np.reshape to achieve this, np.reshape is slow if the volume gets large (> 100x100x100), which I'm not sure why. I thought I can use as_stride to reshape the output, but numpy crashes (code snippet below). Any ideas on how I can get a view of the output from the helper method as N**3x1xnxnxn without using np.reshape?

import numpy as np
import skimage
l = 15
s = 3
X = np.ones((l,l,l))      
print('actual shape',X.shape)
view = skimage.util.shape.view_as_blocks(X,(s,s,s))    
print('original view',view.shape)

new_shape = ((l/s)**3,1,1,s,s,s)
print('new view',new_shape)    

view_correct = view.reshape(new_shape)
print(view_correct.shape)
print('coord:','124,0,0,2,2,2','value:',view_correct[124,0,0,2,2,2])

view_incorrect = np.lib.stride_tricks.as_strided(view, shape=new_shape)
print(view_incorrect.shape)
print('coord:','124,0,0,2,2,2','value:',view_incorrect[124,0,0,2,2,2])
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

I took an example from view_as_blocks, and tried your style of reshape:

A = np.arange(4*4).reshape(4,4)
B = view_as_blocks(A, block_shape=(2, 2))
print(A.__array_interface__)
print(B.__array_interface__)

C = B.reshape((2*2,2,2))
print(C.__array_interface__)

producing:

{'typestr': '<i4', 'data': (153226600, False), 'shape': (4, 4),
 'descr': [('', '<i4')], 'version': 3, 'strides': None}
{'typestr': '<i4', 'data': (153226600, False), 'shape': (2, 2, 2, 2),
 'descr': [('', '<i4')], 'version': 3, 'strides': (32, 8, 16, 4)}
{'typestr': '<i4', 'data': (150895960, False), 'shape': (4, 2, 2),
 'descr': [('', '<i4')], 'version': 3, 'strides': None}

The data pointer for A and B is the same; B is a view on A.

But the pointer for C is different. It is a copy. That explains why it takes so long in your case.


Lets do that a little differently:

A = np.arange(4*4).reshape(4,4)
B = view_as_blocks(A, block_shape=(2, 2))
print(A.__array_interface__)
print(B.__array_interface__)

C = B.reshape((2*2,1,2,2))
print(C.__array_interface__)

D = as_strided(B, shape=(2*2,1,2,2))
print(D.__array_interface__)

print(B[1,1,:,:])
print(C[3,0,:,:])
print(D[3,0,:,:])

producing

1254:~/mypy$ python3 skshape.py 
{'strides': None, 'typestr': '<i4', 'version': 3, 
 'data': (154278744, False), 'shape': (4, 4), 'descr': [('', '<i4')]}
{'strides': (32, 8, 16, 4), 'typestr': '<i4', 'version': 3, 
 'data': (154278744, False), 'shape': (2, 2, 2, 2), 'descr': [('', '<i4')]}
{'strides': None, 'typestr': '<i4', 'version': 3, 
 'data': (155705400, False), 'shape': (4, 1, 2, 2), 'descr': [('', '<i4')]}
{'strides': (32, 8, 16, 4), 'typestr': '<i4', 'version': 3, 
 'data': (154278744, False), 'shape': (4, 1, 2, 2), 'descr': [('', '<i4')]}

[[10 11]
 [14 15]]
[[10 11]
 [14 15]]
[[  154561960 -1217783696]
 [         48        3905]]

Again the reshape creates a copy. The 2nd as_strides returns a view, but the striding is screwed up. It is looking at memory outside the original data buffer (that's part of why playing with strides on your own is dangerous).


In my example, look at the first corner value of each block

print(B[:,:,0,0])
print(C[:,0,0,0])

[[ 0  2]
 [ 8 10]]
[ 0  2  8 10]

For B, the rows increase by 8, columns by 2; that's reflected in the (32,8) (4*8,4*2) striding.

But in C the steps are (2,6,2) - striding can't do that.

From this I conclude that the reshape is impossible without copy.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...