javascript - Remove Duplicate objects from JSON Array

Question

I have an array that looks like this:

var standardsList = [
    {"Grade": "Math K", "Domain": "Counting & Cardinality"},
    {"Grade": "Math K", "Domain": "Counting & Cardinality"},
    {"Grade": "Math K", "Domain": "Counting & Cardinality"},
    {"Grade": "Math K", "Domain": "Counting & Cardinality"},
    {"Grade": "Math K", "Domain": "Geometry"},
    {"Grade": "Math 1", "Domain": "Counting & Cardinality"},
    {"Grade": "Math 1", "Domain": "Counting & Cardinality"},
    {"Grade": "Math 1", "Domain": "Orders of Operation"},
    {"Grade": "Math 2", "Domain": "Geometry"},
    {"Grade": "Math 2", "Domain": "Geometry"}
];

And I need to remove the duplicates so that something like this remains:

var standardsList = [
    {"Grade": "Math K", "Domain": "Counting & Cardinality"},
    {"Grade": "Math K", "Domain": "Geometry"},
    {"Grade": "Math 1", "Domain": "Counting & Cardinality"},
    {"Grade": "Math 1", "Domain": "Orders of Operation"},
    {"Grade": "Math 2", "Domain": "Geometry"}
];

I've tried installing underscore.js and using ._uniq but that only seems to work when a single key:value pair appears in the object. I can't seem to get it to work across multiple keys.

When I try something like:

var uniqueStandards = _.uniq(standardsList, function(item, key, Domain){
    return item.Domain;
});

I only get the first three unique values (one per grade). But I need all the unique values across both grade and domain. Is there a simple way to feed both keys to the _.uniq function?

Ultimately, I need a list with the each unique grade as the header and the unique domains as the list items to pass into an HTML page. I may be going about this wrong, so if there is an easier way to accomplish that end goal, I am open to ideas.

Thanks in advance!

Edit: Getting some good responses and wanted to clarify what my end goal was. I'm trying to create a series of lists in HTML of the form:

<div>
    <h3>Math K</h3>
    <li>Counting & Cardinality</li>
    <li>Geometry</li>
</div>
<div>
    <h3>Math 1</h3>
    <li>Counting & Cardinality</li>
    <li>Orders of Operation</li>
</div>
<div>
    <h3>Math 2</h3>
    <li>Geometry</li>
</div>

My original though was to create an array and push that into the <div> element on the page with $("#divid").append(array)

question from:https://stackoverflow.com/questions/23507853/remove-duplicate-objects-from-json-array

Answer 1

Ultimately, I need a list with the each unique grade as the header and the unique domains as the list items to pass into an HTML page. I may be going about this wrong, so if there is an easier way to accomplish that end goal, I am open to ideas.

So you don't actually need that output array in the format you asked about.

That being the case, I would cut directly to the chase with a very simple and efficient solution:

var grades = {};
standardsList.forEach( function( item ) {
    var grade = grades[item.Grade] = grades[item.Grade] || {};
    grade[item.Domain] = true;
});

console.log( JSON.stringify( grades, null, 4 ) );

The resulting grades object is:

{
    "Math K": {
        "Counting & Cardinality": true,
        "Geometry": true
    },
    "Math 1": {
        "Counting & Cardinality": true,
        "Orders of Operation": true
    },
    "Math 2": {
        "Geometry": true
    }
}

One interesting thing about this approach is that it is very fast. Note that it makes only a single pass through the input array, unlike other solutions that require multiple passes (whether you write them yourself or whether _.uniq() does it for you). For a small number of items this won't matter, but it's good to keep in mind for larger lists.

And with this object you now have everything you need to run any code or generate any other format you want. For example, if you do need the exact array output format you mentioned, you can use:

var outputList = [];
for( var grade in grades ) {
    for( var domain in grades[grade] ) {
        outputList.push({ Grade: grade, Domain: domain });
    }
}

JSON.stringify( outputList, null, 4 );

This will log:

[
    {
        "Grade": "Math K",
        "Domain": "Counting & Cardinality"
    },
    {
        "Grade": "Math K",
        "Domain": "Geometry"
    },
    {
        "Grade": "Math 1",
        "Domain": "Counting & Cardinality"
    },
    {
        "Grade": "Math 1",
        "Domain": "Orders of Operation"
    },
    {
        "Grade": "Math 2",
        "Domain": "Geometry"
    }
]

Rai asks in a comment how this line of code works:

var grade = grades[item.Grade] = grades[item.Grade] || {};

This is a common idiom for fetching an object property or providing a default value if the property is missing. Note that the = assignments are done in right-to-left order. So we could translate it literally to use an if statement and a temp variable:

// Fetch grades[item.Grade] and save it in temp
var temp = grades[item.Grade];
if( ! temp ) {
    // It was missing, so use an empty object as the default value
    temp = {};
}
// Now save the result in grades[item.Grade] (in case it was missing)
// and in grade
grades[item.Grade] = temp;
var grade = temp;

You may notice that in the case where grades[item.Grade] already exists, we take the value we just fetched and store it back into the same property. This is unnecessary, of course, and you probably wouldn't do it if you were writing the code out like this. Instead, you would simplify it:

var grade = grades[item.Grade];
if( ! grade ) {
    grade = grades[item.Grade] = {};
}

That would be a perfectly reasonable way to write the same code, and it's more efficient too. It also gives you a way to do a more specific test than the "truthiness" that the || idiom relies on. For example instead of if( ! grade ) you might want to use if( grade === undefined ).