How to get a better performance to load ElasticSearch data into Hive?

Question

We created the Hive external table using ElasticSearch StorageHandler as shown below:

CREATE EXTERNAL TABLE DEFAULT.ES_TEST (
  REG_DATE STRING
, STR1     STRING
, STR2     STRING
, STR3     STRING
, STR4     STRING
, STR5     STRING
)
ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe'
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES (
  'es.resource'          = 'log-22-20210120'
, 'es.nodes'             = '1.2.3.4'
, 'es.port'              = '9201'
, 'es.mapping.date.rich' = 'false'
);

And then we tried to load the ES data into Hive managed table as like:

insert overwrite table elastic.es_log_tab partition(part_log_date)
select *
,      current_timestamp()
,      from_unixtime(unix_timestamp(reg_date), 'yyyyMMdd')
from   DEFAULT.ES_TEST;

When the ES data for given date is volumed to about 65GB, it was approximated taken 10 hours or 1.1M rows per minute (670M rows in total).

In order to get a better loading performance for this case, are there any further recommendation or checkpoints? How about increasing the number of mappers? Currently, it is running with 16 mappers. Expecting to get it faster with more mappers?

Please share your thoughts and previous experience with me.

question from:https://stackoverflow.com/questions/65838266/how-to-get-a-better-performance-to-load-elasticsearch-data-into-hive

Answer 1

Waitting for answers