python - Merge csv files - long processing time

Question

I have folder with multiple (hundreds to low thousands) csv files, each with about 150 rows and 2 columns, I need to merge them to 1 summary file to later use for plotting of my data. - each file = trajectory curve

At the moment I use pandas concat

df=pd.DataFrame()
for file in folder:
    df_temp = pd.read_csv(file, skiprows = 10, usecols = [2,3])
    df = pd.concat([df, df_temp], ignore_index = True)

Issue I have now is long processing time (10+ minutes) and occasional MemoryError.

Is there some less memory intensive and reasonably way to merge csv files?

question from:https://stackoverflow.com/questions/65951076/merge-csv-files-long-processing-time

Answer 1

You can create list of DataFrames first and then pass to concat:

dfs=[]
for file in folder:
    dfs.append(pd.read_csv(file, skiprows = 10, usecols = [2,3]))
df = pd.concat(dfs, ignore_index = True)

Solution with list comprehension:

dfs = [pd.read_csv(file, skiprows = 10, usecols = [2,3]) for file in folder]
df = pd.concat(dfs, ignore_index = True)