python - Filling missing values using forward and backward fill in pandas dataframe (ffill and bfill)

Question

Beginner with panda dataframes. I have this data set below with missing values for column A and B (Test.csv):

DateTime              A             B
01-01-2017 03:27        
01-01-2017 03:28        
01-01-2017 03:29    0.18127718  -0.178835737
01-01-2017 03:30    0.186923018 -0.183260853
01-01-2017 03:31        
01-01-2017 03:32        
01-01-2017 03:33    0.18127718  -0.178835737

I can use this code to fill in values using forward propagation, but this only fills in for 03:31 and 03:32, and not 03:27 and 03:28.

import pandas as pd
import numpy as np

df = pd.read_csv('test.csv', index_col = 0)
data = df.fillna(method='ffill')
ndata = data.to_csv('test1.csv')

results in:

   DateTime              A             B
    01-01-2017 03:27        
    01-01-2017 03:28        
    01-01-2017 03:29    0.18127718  -0.178835737
    01-01-2017 03:30    0.186923018 -0.183260853
    01-01-2017 03:31    0.186923018 -0.183260853
    01-01-2017 03:32    0.186923018 -0.183260853
    01-01-2017 03:33    0.18127718  -0.178835737

How could I include the 'Bfill' to fill in the missing values for 03:27 and 03:28 using the backfil?

Answer 1

You can use ffill and bfill if need replace NaN values forward and backward filling:

print (df)
                         A         B
DateTime                            
01-01-2017 03:27       NaN       NaN
01-01-2017 03:28       NaN       NaN
01-01-2017 03:29  0.181277 -0.178836
01-01-2017 03:30  0.186923 -0.183261
01-01-2017 03:31       NaN       NaN
01-01-2017 03:32       NaN       NaN
01-01-2017 03:33  0.181277 -0.178836

data = df.ffill().bfill()
print (data)
                         A         B
DateTime                            
01-01-2017 03:27  0.181277 -0.178836
01-01-2017 03:28  0.181277 -0.178836
01-01-2017 03:29  0.181277 -0.178836
01-01-2017 03:30  0.186923 -0.183261
01-01-2017 03:31  0.186923 -0.183261
01-01-2017 03:32  0.186923 -0.183261
01-01-2017 03:33  0.181277 -0.178836

Which is same as the function fillna with parameters:

data = df.fillna(method='ffill').fillna(method='bfill')