tensorflow - Machine Learning CTC loss log probability

Question

I have a trained CRNN model which is supposed to recognise text from images. It really works and so far so good.

My output is a CTC loss layer and I decode it with the tensorflow function keras.backend.ctc_decode which returns, as the documentations says (https://code.i-harness.com/en/docs/tensorflow~python/tf/keras/backend/ctc_decode), a Tuple with the decoded result and a Tensor with the log probability of the prediction.

By making some tests with the model, I get this results:

True value: test0, prediction: test0, log_p: 1.841524362564087
True value: test1, prediction: test1, log_p: 0.9661365151405334
True value: test2, prediction: test2, log_p: 1.0634151697158813
True value: test3, prediction: test3, log_p: 2.471940755844116
True value: test4, prediction: test4, log_p: 1.4866207838058472
True value: test5, prediction: test5, log_p: 0.7630811333656311
True value: test6, prediction: test6, log_p: 0.35642576217651367
True value: test7, prediction: test7, log_p: 1.5693446397781372
True value: test8, prediction: test8, log_p: 0.9700028896331787
True value: test9, prediction: test9, log_p: 1.4783780574798584

The prediction is always correct. However what I think it's the probability seems not to be what I expect. They looks like completely random numbers, even grater than 1 or 2! What am I doing wrong??

question from:https://stackoverflow.com/questions/65647571/machine-learning-ctc-loss-log-probability

Answer 1

Well, I guess you've mixed up Probability and Log Probability together. While your intuition is correct that the probability value anything above or below 0-1 would have been weird. However your function isn't giving you the probabilities but the log probabilities, which is actually nothing but the probability in the logarithmic scale. So everything's good with your model.

If you're wondering why we work with log probabilities instead of probability itself, it has got to do mostly with the scaling issue, however, you could read the thread here

Example on changing Log Probabilities into Actual Probabilities:

import numpy as np

# some random log probabilities
log_probs = [-8.45855173, -7.45855173, -6.45855173, -5.45855173, -4.45855173, -3.45855173, -2.45855173, -1.45855173, -0.45855173]

# Let's turn these into actual probabilities (NOTE: If you've "negative" log probabilities, then simply negate the exponent, like np.exp(-x))
probabilities = np.exp(log_probs)

print(probabilities)

# Output:
[2.12078996e-04, 5.76490482e-04, 1.56706360e-03, 4.25972051e-03, 1.15791209e-02, 3.14753138e-02, 8.55587737e-02, 2.32572860e-01, 6.32198578e-01] # everything is between [0-1]