tensorflow - Keras model generates Nan on predict on gcp

Question

I have a Keras model (Yolo3) and I want to run the code on several images in the loop:

I am using Debian 10, Tensorflow 2.4, cuda 11 and K80 Tesla GPU on GCP virtual machine.

here is the code on colab code

I remark that the code is running very well on my local GPU using RTX2070 and tensorflow 2.3

Running exactly the same code on GCP instance crash at the second iteration by predicting NaN value !!!!)

I debug the image (infinite, Nan value), consider a list of the same image, etc, it still giving the same error!!

I also debug the weights of the model and I get that it has the same weights across iterations!

Anyone from the GCP team can help ?

question from:https://stackoverflow.com/questions/65834183/keras-model-generates-nan-on-predict-on-gcp

Answer 1

Waitting for answers