pandas - How to concatenate all CSVs in a directory, adding CSV name as a column with Python

Question

• I have a folder with about 100 CSVs (Downloads/challenges).
• Each CSV has the same 50+ columns.
• Each CSV is titled something like azerbaijan_challenge_entrants.csv.

I want to create one new CSV (all_entrants.csv) that includes all data from all 100 CSVs, adding one new column: challenge, which should include the name of the CSV that the row data came from.

I generally like Python for tasks like this. But I am struggling to make this work. Any help would be appreciated!

Answer 1

This is possible with os from the standard library and 3rd party library pandas:

import os
import pandas as pd

mypath = os.path.join('Downloads', 'challenges')

# get list of files
files = [f for f in os.listdir(mypath) if os.path.isfile(os.path.join(mypath, f))]

# build list of dataframes, adding "challenge" column
dfs = [pd.read_csv(os.path.join(mypath, f)).assign(challenge=f) for f in files]

# concatenate dataframes into one
df = pd.concat(dfs, ignore_index=True)

# write to csv
df.to_csv('all_entrants.csv')