Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
491 views
in Technique[技术] by (71.8m points)

python - Buffered data was truncated after reaching the output size limit

When I use Colaboratory to run my NIN model, it occurs an error in the output of training process which tells "Buffered data was truncated after reaching the output size limit." in the 61th epoch. I have no idea about this crash.Is my code still running ? How can I solve this problem?

Here is some information about the output of my training process:

Epoch 57/200
391/391 [==============================] - 53s 135ms/step - loss: 0.8365 - acc: 0.7784 - val_loss: 0.9250 - val_acc: 0.7625
Epoch 58/200
 28/391 [=>............................] - ETA: 46s - loss: 0.8356 - acc: 0.7835391/391 [==============================] - 53s 136ms/step - loss: 0.8288 - acc: 0.7811 - val_loss: 0.8977 - val_acc: 0.7608
Epoch 59/200
326/391 [========================>.....] - ETA: 8s - loss: 0.8309 - acc: 0.7789391/391 [==============================] - 53s 136ms/step - loss: 0.8297 - acc: 0.7798 - val_loss: 0.9030 - val_acc: 0.7628
Epoch 60/200
391/391 [==============================] - 53s 134ms/step - loss: 0.8245 - acc: 0.7825 - val_loss: 0.8378 - val_acc: 0.7767
Epoch 61/200
 28/391 [=>............................] - ETA: 46s - loss: 0.8281 - acc: 0.7879390/391 [============================>.] - ETA: 0s - loss: 0.8177 - acc: 0.7851Buffered data was truncated after reaching the output size limit.
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Even if RAM | GPU | DISK on colab is free, this error still comes because there is a limited memory for displaying output of a cell on colab. Assuming the memory limit is around 2Mb to 5Mb when we run many epochs(148+) during training, it tends to fill that memory and hence the output is truncated because there is no more memory left free to display the buffered epochs. However, the machine keeps running in the background and the output is processed but it is not displayed because of the buffered limit. You will still get your desired output.

One solution is not to use verbose=1 (use 0 instead).


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...