python - Removing duplicates from a large image dataset

Question

I am working with a training data set of 127.000 images scraped from internet.

I know there are quite a few duplicates in there and i want to remove them to improve the performance of my deep learning model.

I have tried several different ways to do this. Some did not work at all, others did just remove a few or way too many.

The last one i tried was this:

import hashlib
import os
import PIL
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
%matplotlib inline
import time
import numpy as np

def file_hash(filepath):
    with open(filepath, 'rb') as f:
        return md5(f.read()).hexdigest()

os.chdir('/content/train')

import hashlib, os
duplicates = []
hash_keys = dict()
for index, filename in  enumerate(os.listdir('.')):  #listdir('.') = current directory
    if os.path.isfile(filename):
        with open(filename, 'rb') as f:
            filehash = hashlib.md5(f.read()).hexdigest()
        if filehash not in hash_keys: 
            hash_keys[filehash] = index
        else:
            duplicates.append((index,hash_keys[filehash]))

for file_indexes in duplicates[:30]:
    try:
    
        plt.subplot(121),plt.imshow(imread(file_list[file_indexes[1]]))
        plt.title(file_indexes[1]), plt.xticks([]), plt.yticks([])

        plt.subplot(122),plt.imshow(imread(file_list[file_indexes[0]]))
        plt.title(str(file_indexes[0]) + ' duplicate'), plt.xticks([]), plt.yticks([])
        plt.show()
    
    except OSError as e:
        continue

for index in duplicates:
    os.remove(file_list[index[0]])
  

This method found 490 duplicates, but i am estimating there are at least a couple of thousand duplicates.

I have also tried imagededup with different methods and thresholds.

pip install imagededup

from imagededup.methods import DHash
method_object = DHash()
duplicates = method_object.find_duplicates_to_remove(image_dir='/content/train', 
                                                     max_distance_threshold=3) 

Last run found 23919 duplicates, and its usually somewhere between 20k and 35k depending on the method and the threshold. This is too many. Running the model removing all these produce a worse result.

Anyone know about a better way to remove duplicate images?

question from:https://stackoverflow.com/questions/65651787/removing-duplicates-from-a-large-image-dataset

Answer 1

Waitting for answers