hadoop - Hive every Insert query creates a new file in Hdfs file system

Question

On every insert query one files gets created with 000000_0_copy* in Hdfs file system.

Is this the default behaviour of hive and Hdfs ?

Is there any concept of compaction if yes then How does the comapaction work?

Answer 1

HDFS is an append only filesystem, meaning to modify (UPDATE/DELETE statements) any portion of a file that is already written, one must rewrite the entire file and replace the old file, or write a new file to insert even a single record.

Compaction isn't an automatic process. You need to write your own code to query one table, then insert into another format like parquet/orc