python - How to remove duplicates from list of dicts?

Question

I have a list of dictionaries in python as follows:

[{'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
 {'category': 'software', 'name': 'irssi', 'version': '1.1.2'},
 {'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}]

(parsing some data txt file)

What I wanna do:

If category and name are the same I wanna leave the first appearance of a package entry and remove the rest, so the final output would look like:

[{'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
{'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}]

How should I achieve this? I tried converting the list of dictionaries to a dictionary and then iterate over it with dict.items() but with no luck.

question from:https://stackoverflow.com/questions/65922531/how-to-filter-list-of-dictionaries

Answer 1

Use itertools.groupby, and take first of each group:

def uniq(lst):
    for _, grp in itertools.groupby(lst, lambda d: (d['category'], d['name'])):
        yield list(grp)[0]
lst = [{'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
       {'category': 'software', 'name': 'irssi', 'version': '1.1.2'},
       {'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}]
print(list(uniq(lst))