tensorflow - Keras model generates Nan on predict on gcp

Question

I have a Keras model (Yolo3) and I want to run the code on several images in the loop:

here is the pseudo-code

model = Model(input, output)
for image in list_images:
    image_processed = processing(image)
    result = model.predict(image_processed)

I remark that the code is running very well on my local GPU as expected.

Running exactly the same code on GCP instance K80 crash at the second iteration by predicting NaN value !!!!) I debug the image (infinite, Nan value), consider a list of the same image, etc, it still giving the same error!!

I also debug the weights of the model and I get that it has the same weights across iterations!

Anyone from the GCP team can help ?

If someone wants to reproduce the error, here is the link github

I try to set an issue tracker as mentioned in the first comment but as you see it does not work because I don't have enough permissions.

Answer 1

等待大神解答