machine learning - Python sklearn show loss values during training

Question

I want check my loss values during the training time so I can observe the loss at each iteration. So far I haven't found an easy way for scikit learn to give me a history of loss values, nor did I find a functionality already within scikit to plot the loss for me.

If there was no way to plot this, it'd be great if I could simply fetch the final loss values at the end of classifier.fit.

Note: I am aware of the fact that some solutions are closed form. I'm using several classifiers which do not have analytical solutions, such as logistic regression and svm.

Does anyone have any suggestions?

Answer 1

So I couldn't find very good documentation on directly fetching the loss values per iteration, but I hope this will help someone in the future:

old_stdout = sys.stdout
sys.stdout = mystdout = StringIO()
clf = SGDClassifier(**kwargs, verbose=1)
clf.fit(X_tr, y_tr)
sys.stdout = old_stdout
loss_history = mystdout.getvalue()
loss_list = []
for line in loss_history.split('
'):
    if(len(line.split("loss: ")) == 1):
        continue
    loss_list.append(float(line.split("loss: ")[-1]))
plt.figure()
plt.plot(np.arange(len(loss_list)), loss_list)
plt.savefig("warmstart_plots/pure_SGD:"+str(kwargs)+".png")
plt.xlabel("Time in epochs")
plt.ylabel("Loss")
plt.close()

This code will take a normal SGDClassifier(just about any linear classifier), and intercept the verbose=1 flag, and will then split to get the loss from the verbose printing. Obviously this is slower but will give us the loss and print it.