Tensorflow object detection API only using CPU and Crashing

Question

Hi everyone i'm trying to use the object detection API from Tensorflow. i'm currently using

TF 2.2.0 and TF-GPU 2.2.0

RTX2080 cudatoolkit v10.1 CUDNN v7.6.5

i'm trying to train the model ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8 on my own Dataset.

when using the model_main_tf2.py ,the training worked but somehow it always crash after 400 steps. and even though i add this line with tf.device(tf.DeviceSpec(device_type="GPU", device_index=0)): if i open Task manager to see the GPU usage, it always says 5% or less and the CPU around 34% after it crash i alway get the same Error:

INFO:tensorflow:Step 500 per-step time 0.531s loss=0.864
I0129 00:07:36.263110 12688 model_lib_v2.py:651] Step 500 per-step time 0.531s loss=0.864
2021-01-29 00:07:58.237349: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_UNKNOWN: unknown error
2021-01-29 00:07:58.258278: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1
Fatal Python error: Aborted

any idea why it could happen?

question from:https://stackoverflow.com/questions/65946644/tensorflow-object-detection-api-only-using-cpu-and-crashing

Answer 1

this is happen when your Ram getting full ,and to solve this ,you need to decrease number of

batchsize in pipeline.config and retriain again

or train on google Colab or upgrade your Ram size.

Regards.