search - ElasticSearch: Index vs type and handling updates

Question

I'm pretty familiar with the capabilities of ElasticSearch and its benefits, but this is the first time I'm getting my hands dirty building an index. So I'm eager to get the following approach vetted with experts.

Requirement:

Our application gets metadata about various products from multiple sources. Typically these feeds come in as XML files (file size can vary from 2 GB to 12 GB and sometimes a single record is spread across multiple files) and the information contained in the feed from one provider may or may not overlap with information contained in others. So we de-dupe this data, normalize this data to a set of common formats, depending on the product type, and we need to provide the ability to search against this consolidated data set (obviously this is where ElasticSearch comes in).

All products have certain common identifiers (like id, price, etc.), but the core metadata can look completely different between different product types. To quantify this, let's say all products have 30% fields in common and 70% fields differ between product types. There aren't too many product types and it's safe to assume that there will not be more than 10 types at any point in time. To start with the number is much smaller (around 3-4).

Additionally, there can be updates coming in at random intervals from these data sources and some of these updates need to be reflected in searches right away (near real time), without bringing down the search capability.

The proposed solution:

I'm considering having different indexes for different product types, based on what I read here: https://www.elastic.co/blog/index-vs-type. So the normalization job will look at the source files, create the normalized structure for the given product type and add it to the appropriate index. The search API that we expose will perform a search against each of these indices for the search term used and consolidate the results into a single JSON response with multiple sections (one section for each product type).

For updates, we plan on using the bulk API for update, insert and delete and given that the limitations around REST API calls, we will have to make these calls in batches of x MB each.

Question:

Is this the best way to organize the data and update it in ElasticSearch (for my use-case)? Would it be better to use multiple types on the same index (example: /products containing products of type typeA, typeB, etc.) instead of creating one index per product type? If so, will the search be significantly faster than searching across indices? Are there better ways to handle the CRUD of records after the index has been created?

Thanks in advance!

Answer 1

First it is worth noting that mapping types will go away in the next ES release (soft deprecation in ES 6 and removal in ES 7).

Now whether types go away or not, it is still possible to use a single index, however you'd increase sparsity since only 30% of your fields are common and that should be avoided at all cost.

So, I'd say that your multi-index approach is the only one that makes sense given the nature of your data.

Additional information worth reading: https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html