amazon web services - Custom training Tensorflow on Sagemaker: WARNING keras.py:603 cannot save layer values during forward pass with tf.function

Question

I am training a custom model with a custom layer using Tensorflow on Sagemaker. It runs smoothly local, no errors, no warnings. When I run it on Sagemaker, it produces many warnings, all the same:

2021-01-22 15:20:26.660 xxxx.xxxx.compute.internal:32 WARNING keras.py:603] cannot save layer values during forward pass with tf.function

I am not sure if this is something to be worried about. Does someone have any ideas about this?

question from:https://stackoverflow.com/questions/65848909/custom-training-tensorflow-on-sagemaker-warning-keras-py603-cannot-save-layer

Answer 1

This warning is logged smdebug/tensorflow/keras.py from changeset 9ad5d8251dfa3cab2be494cd49f06678a6a84a4e, line 603 from 2020-12-08.

In changeset da3b4e30b09dda4813c7b6b3bef0e7fb0c8243cf, line 604 from 2020-12-23 the warning was removed, so apparently the authors decided the warning is redundant.