Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
566 views
in Technique[技术] by (71.8m points)

python - WARNING:tensorflow:Layer my_model is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2

Before my Tensorflow neural network starts training, the following warning prints out:

WARNING:tensorflow:Layer my_model is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx. If you intended to run this layer in float32, you can safely ignore this warning.

If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2. To change all layers to have dtype float64 by default, call tf.keras.backend.set_floatx('float64').

To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

Now, based on the error message, I am able to silence this error message by setting the backend to 'float64'. But, I would like to get to the bottom of this and set the right dtypes manually.

Full code:

import tensorflow as tf
from tensorflow.keras.layers import Dense, Concatenate
from tensorflow.keras import Model
from sklearn.datasets import load_iris
iris, target = load_iris(return_X_y=True)

X = iris[:, :3]
y = iris[:, 3]

ds = tf.data.Dataset.from_tensor_slices((X, y)).shuffle(25).batch(8)

class MyModel(Model):
  def __init__(self):
    super(MyModel, self).__init__()
    self.d0 = Dense(16, activation='relu')
    self.d1 = Dense(32, activation='relu')
    self.d2 = Dense(1, activation='linear')

  def call(self, x):
    x = self.d0(x)
    x = self.d1(x)
    x = self.d2(x)
    return x

model = MyModel()

loss_object = tf.keras.losses.MeanSquaredError()

optimizer = tf.keras.optimizers.Adam(learning_rate=5e-4)

loss = tf.keras.metrics.Mean(name='loss')
error = tf.keras.metrics.MeanSquaredError()

@tf.function
def train_step(inputs, targets):
    with tf.GradientTape() as tape:
        predictions = model(inputs)
        run_loss = loss_object(targets, predictions)
    gradients = tape.gradient(run_loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    loss(run_loss)
    error(predictions, targets)

for epoch in range(10):
  for data, labels in ds:
    train_step(data, labels)

  template = 'Epoch {:>2}, Loss: {:>7.4f}, MSE: {:>6.2f}'
  print(template.format(epoch+1,
                        loss.result(),
                        error.result()*100))
  # Reset the metrics for the next epoch
  loss.reset_states()
  error.reset_states()
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

tl;dr to avoid this, cast your input to float32

X = tf.cast(iris[:, :3], tf.float32) 
y = tf.cast(iris[:, 3], tf.float32)

or with numpy:

X = np.array(iris[:, :3], dtype=np.float32)
y = np.array(iris[:, 3], dtype=np.float32)

Explanation

By default, Tensorflow uses floatx, which defaults to float32, which is standard for deep learning. You can verify this:

import tensorflow as tf
tf.keras.backend.floatx()
Out[3]: 'float32'

The input you provided (the Iris dataset), is of dtype float64, so there is a mismatch between Tensorflow's default dtype for weights, and the input. Tensorflow doesn't like that, because casting (changing the dtype) is costly. Tensorflow will generally throw an error when manipulating tensors of different dtypes (e.g., comparing float32 logits and float64 labels).

The "new behavior" it's talking about:

Layer my_model_1 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2

Is that it will automatically cast the input dtype to float32. Tensorflow 1.X probably threw an exception in this situation, although I can't say I've ever used it.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...