python - import CSV file, remove a line, export CSV file

Question

I'm trying to import, modify and then export a CSV file for doing bulk delivery's.

essentially, i would like to open the file, remove a specific column from it, check for any duplicates and remove if there are any and then export to another CSV file.

my code is as follows

import csv

with open ('book1.csv', 'r') as in_file,  open ('ammended.csv', 'w') as out_file:
    read_file = csv.DictReader(in_file)

    for row in read_file:
        print(row)
        row.pop('flavour')
        print(row) 

output -

{'name': 'me', 'address': '34', 'postcode': 'hhhh', 'flavour':'lemon'}
{'name': 'me', 'address': '34', 'postcode': 'hhhh'}
{'name':'me', 'address': '34', 'postcode': 'hhhh', 'flavour': 'lime'}
{'name':'me', 'address': '34', 'postcode': 'hhhh'}
{'name': 'you', 'address':'35', 'postcode': 'hlhl', 'flavour': 'strawberry'} 
{'name': 'you','address': '35', 'postcode': 'hlhl'}

so far im happy that it imports and removes the correct line, but then how can i check for duplicate lines and then export to a new file.

question from:https://stackoverflow.com/questions/65916988/import-csv-file-remove-a-line-export-csv-file

Answer 1

I would suggest you trying pandas for your task.

To install pandas type in your terminal:

pip install pandas

In your code try doing:

import pandas as pd

df = pd.read_csv('book1.csv')   # Reading data
df = df.drop('flavour', axis=1) # Dropping column
df = df.drop_duplicates()       # Removing duplicates
df.to_csv('ammended.csv')       # Saving filtered data