python - reading images in a loop using requests.get changes behavior

Question

I need to read thousends of images hosted in a sharepoint site, I save their urls in a list, and then read them in a loop with the following code:

def url_to_image(url):
    resp = requests.get(url).raw
    image = np.asarray(bytearray(resp.read()), dtype="uint8")
    image = cv2.imdecode(image, cv2.IMREAD_COLOR)
return image

Everything goes well, except that speed changes drastically between different iterations. the behavior I was able to spot is that I can read and process ~20 images in a row with a speed of 0-3 secs per image which is good enough, but then the next ~10 images take very long, 10-35 secs per image, and so it repeats itself over and over again. My first thought was that it has to do with image size, but I was running the code a few times on the same data of images, and it had different behavior each time.

I checked timing for each part of the code, and it looks like the part that gets stuck is the read() of the response.

I tried also in a different method, using the sharepoint rest api directly, but the behavior was similar.

To my knowledge there are no server or connection problem, as I make a bunch of other requests with no problem at all.

Any help will be much appreciated.

question from:https://stackoverflow.com/questions/65873081/reading-images-in-a-loop-using-requests-get-changes-behavior

Answer 1

Waitting for answers