mongodb - mongo 3 duplicates on unique index - dropDups

Question

In the documentation for mongoDB it says: "Changed in version 3.0: The dropDups option is no longer available."

Is there anything I can do (other than downgrading) if I actually want to create a unique index and destroy duplicate entries?

please keep in mind the I receive about 300 inserts per second so I can't just delete all duplicates and hope none will come in by the time I'm done indexing.

Answer 1

Yes dropDupes is now deprecated since version 2.7.5 because it was not possible to predict correctly which document would be deleted in the process.

Typically, you have 2 options :

1. Use a new collection :

   • Create a new collection,
   • Create the unique index on this new collection,
   • Run a batch to copy all the documents from the old collection to the new one and make sure you ignore duplicated key error during the process.

2. Deal with it in your own collection manually :

   • make sure you won't insert more duplicated documents in your code,
   • run a batch on your collection to delete the duplicates (and make sure you keep the good one if they are not completely identical),
   • then add the unique index.

For your particular case, I would recommend the first option but with a trick :

• Create a new collection with unique index,
• Update your code so you now insert documents in both tables,
• Run a batch to copy all documents from the old collection to the new one (ignore duplicated key error),
• rename the new collection to match the old name.
• re-update your code so you now write only in the "old" collection