performance - Determining index each group duplicate values in an array in Python with the fastest way

Question

I want to find an index of each group duplicate value like this:

s = [2,6,2,88,6,...]

The results must return the index from original s: [[0,2],[1,4],..] or the result can show another way.

I find many solutions so I find the fastest way to get duplicate group:

s = np.sort(a, axis=None)
s[:-1][s[1:] == s[:-1]]

But after sort I got wrong index from original s.

In my case, I have ~ 200mil value on the list and I want to find the fastest way to do that. I use an array to store value because I want to use GPU to make it faster.

question from:https://stackoverflow.com/questions/65878826/determining-index-each-group-duplicate-values-in-an-array-in-python-with-the-fas

Answer 1

Using hash structures like dict helps.

For example:

import numpy as np
from collections import defaultdict

a=np.array([2,4,2,88,15,4])
table=defaultdict(list)
for ind,num in enumerate(a):
    table[num]+=[ind]

Outputs:

{2: [0, 2], 4: [1, 5], 88: [3], 15: [4]}

If you want to show duplicated elements in the order from small to large:

for k,v in sorted(table.items()):
    if len(v)>1:
        print(k,":",v)

Outputs:

2 : [0, 2]
4 : [1, 5]

The speed is determined by how many different values in the number list.