python - How to fix nan values coming from the implementation of Logistic Regression?

Question

After I get some processes to get x_train and y_train, I flatten them. The code snippets are shown below.

The flatten code

x_train = x_train_flatten.T
x_test = x_test_flatten.T
y_test = Y_test.T
y_train = Y_train.T
print("x train: ",x_train.shape)
print("x test: ",x_test.shape)
print("y train: ",y_train.shape)
print("y test: ",y_test.shape)

Result

x train:  (16384, 38)
x test:  (16384, 10)
y train:  (1, 38)
y test:  (1, 10)

Normalization Process(AfterFlatten) :

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
imputer = imputer.fit(x_train)
x_train= imputer.transform(x_train)
x_train = (x_train-np.min(x_train))/(np.max(x_train)-np.min(x_train))

Then I wrote some methods for Logistic Regression

Logistic Regression Methods

def initialize_weights_and_bias(dimension):
    w = np.full((dimension,1),0.01)
    b = 0.0
    return w, b

def sigmoid(z):
    y_head = 1/(1+np.exp(-z))
    return y_head

def forward_backward_propagation(w,b,x_train,y_train):
    # forward propagation
    z = np.dot(w.T,x_train) + b    
    y_head = sigmoid(z)    
    loss = -(1-y_train)*np.log(1-y_head)-y_train*np.log(y_head)        
    cost = (np.sum(loss))/x_train.shape[1]  # x_train.shape[1]  is for scaling
    
    # backward propagation
    derivative_weight = (np.dot(x_train,((y_head-y_train).T)))/x_train.shape[1] # x_train.shape[1]  is for scaling
    derivative_bias = np.sum(y_head-y_train)/x_train.shape[1]                   # x_train.shape[1]  is for scaling
    gradients = {"derivative_weight": derivative_weight,"derivative_bias": derivative_bias}
    return cost,gradients

def update(w, b, x_train, y_train, learning_rate,number_of_iterarion):
    cost_list = []
    cost_list2 = []
    index = []
    # updating(learning) parameters is number_of_iterarion times
    for i in range(number_of_iterarion):
        # make forward and backward propagation and find cost and gradients
        cost,gradients = forward_backward_propagation(w,b,x_train,y_train)
        cost_list.append(cost)
        # lets update
        w = w - learning_rate * gradients["derivative_weight"]
        b = b - learning_rate * gradients["derivative_bias"]
        if i % 50 == 0:
            cost_list2.append(cost)
            index.append(i)
            print ("Cost after iteration %i: %f" %(i, cost))
    # we update(learn) parameters weights and bias
    parameters = {"weight": w,"bias": b}
    plt.plot(index,cost_list2)
    plt.xticks(index,rotation='vertical')
    plt.xlabel("Number of Iterarion")
    plt.ylabel("Cost")
    plt.show()
    return parameters, gradients, cost_list

def predict(w,b,x_test):
    # x_test is a input for forward propagation
    z = sigmoid(np.dot(w.T,x_test)+b)
    Y_prediction = np.zeros((1,x_test.shape[1]))
    # if z is bigger than 0.5, our prediction is woman (y_head=1),
    # if z is smaller than 0.5, our prediction is man (y_head=0),
    for i in range(z.shape[1]):
        if z[0,i]<= 0.5:
            Y_prediction[0,i] = 0
        else:
            Y_prediction[0,i] = 1
    return Y_prediction

def logistic_regression(x_train, y_train, x_test, y_test, learning_rate ,  num_iterations):
    # initialize
    dimension =  x_train.shape[0] 
    w,b = initialize_weights_and_bias(dimension)
    parameters, gradients, cost_list = update(w, b, x_train, y_train, learning_rate,num_iterations)
    
    y_prediction_test = predict(parameters["weight"],parameters["bias"],x_test)
    y_prediction_train = predict(parameters["weight"],parameters["bias"],x_train)

    train_acc_lr = round((100 - np.mean(np.abs(y_prediction_train - y_train)) * 100),2)
    test_acc_lr = round((100 - np.mean(np.abs(y_prediction_test - y_test)) * 100),2)
    # Print train/test Errors
    print("train accuracy: %", train_acc_lr)
    print("test accuracy: %", test_acc_lr)

Then I start to call logistic_regression method to implement Logistic Regression.

logistic_regression(x_train, y_train, x_test, y_test,learning_rate = 0.01, num_iterations = 700)

After showing some cost results, some of them has nan values as shown below.

Cost after iteration 0: nan
Cost after iteration 50: 10.033753
Cost after iteration 100: 11.253421
Cost after iteration 150: nan
Cost after iteration 200: nan
Cost after iteration 250: nan
Cost after iteration 300: nan
Cost after iteration 350: nan
Cost after iteration 400: nan
Cost after iteration 450: 0.321755
...

How can I fix the issue?

Answer 1

Here is my solution

• Apply feature scaling to x_train before training the model to stop producing nan values

I write this code block before calling logistic_regression method.

from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
x_train = sc_X.fit_transform(x_train)