python - Adding a line-terminator in pandas ends up adding another

Question

I am able to load a csv file fine into a pandas dataframe with the panda defaults:

df = pd.read_csv(file)

>>> df
   distance  recession_velocity
0   # not a row                 NaN
1         0.032               170.0
2         0.034               290.0
3         0.214              -130.0

However, as soon as I add the lineterminator, the program seems to go haywire:

df = pd.read_csv(file, lineterminator='
')
       distance recession_velocity

0   # not a row                   

1         0.032                170

2         0.034                290

3         0.214               -130

The file indeed does have a line separator:

>>> print(repr(open('/Users/david/example.csv').read()))
'distance,recession_velocity
# not a row,
0.032,170
0.034,290
0.214,-130
0.263,

What is the issue here and is there a way to fix it without having to trim all the column values?

Answer 1

Python's file objects will automatically translate to in text mode. read_csv uses its own file handling, it will indeed see instead, so if you pass lineterminator=" " it will really just trim that one character.

If you don't pass the lineterminator parameter at all, it will guess the line-ending style. You can also pass in a file object instead of a path. This may slow things down a bit, but it will give you the same transformation behaviour that you see when you do a straight read.