python - csv.writer.writerow with a comma in a field not writing correctly

Question

I am pulling data from a website and saving it to CSV in Python. Works most of the time except when a specific field in one of the rows occasionally is in the thousands and the vendor includes a comma in the field. No matter how I use writer.writerow() I cant seem to exclude this comma from being interpreted as the delimeter EVEN though the vendor enclosed the specific field in double quotes when the comma appears.

# Obtain the data from the website
downloadresponse = s.post(downloadurl, data=downloaddata,headers=postheaders, cookies=loginresponse.cookies)
print("Download response status =", downloadresponse.status_code)
print("Downloading " +filename+" now.")

with open(filename, 'w') as f:
    writer = csv.writer(f, delimiter =',',escapechar='\' ,quotechar ='"', quoting=csv.QUOTE_MINIMAL, doublequote=True)
    for line in downloadresponse.iter_lines():
        print("Raw Line =", line)
        print("UTF Line =", line.decode('utf-8'))
        print("UTF & Split Line =", line.decode('utf-8').split(','))
        writer.writerow(line.decode('utf-8').split(','))

Produces this output for the problematic row in the CSV.

Download response status = 200 
Download response reason = OK 
Downloading /Users/someone/Documents/@Investing/UOA-Workspace/data/uoa-eod-02-04-2021.csv now.

DEBUG-Raw Line = b'AMC,7.31,Call,2,02/05/21,1,5.25,5.33,5.4,5.3,739,283,2.61,"1,246.65%","15:10 ET"' 
DEBUG-UTF Line = AMC,7.31,Call,2,02/05/21,1,5.25,5.33,5.4,5.3,739,283,2.61,"1,246.65%","15:10 ET" 
DEBUG-UTF & Split Line = ['AMC', '7.31', 'Call', '2', '02/05/21', '1', '5.25', '5.33', '5.4', '5.3', '739', '283', '2.61', '"1', '246.65%"', '"15:10 ET"']

So the field "1,246.65%" in the raw LINE is being split on the comma into two fields '"1', '246.65%"' even though it's contained in double quotes. Can anyone see what I am doing wrong?

question from:https://stackoverflow.com/questions/66053420/csv-writer-writerow-with-a-comma-in-a-field-not-writing-correctly

Answer 1

As mentioned in the comments, you can write directly, without using csv module. Line looks properly quoted. Something like

with open(filename, 'w') as f:
    f.write(downloadresponse.text)

This may need some tweaks. e.g. in case of streaming response you may need to write line by line.