Remove duplicates python/pandas

Question

I have data like this:

d = {'col1': ['1-2-3', '2-2-3'], 'col2': ['1-2-2-3', '4-4-4']}
df = pd.DataFrame(data=d)
df

    col1    col2
0   1-2-3   1-2-2-3
1   2-2-3   4-4-4

I need to remove the duplicates and preserve the order to achieve this:

    col1    col2
0   1-2-3   1-2-3
1   2-3     4

question from:https://stackoverflow.com/questions/65842080/remove-duplicates-python-pandas

Answer 1

Use DataFrame.applymap...

if order is important use split with dict.fromkeys for remove duplicates, last join back:

df = df.applymap(lambda x: '-'.join(dict.fromkeys(x.split('-')).keys()))
print (df)
    col1   col2
0  1-2-3  1-2-3
1    2-3      4

if order is not important use sets:

df = df.applymap(lambda x: '-'.join(set(x.split('-'))))
print (df)
    col1   col2
0  3-2-1  3-2-1
1    3-2      4