scala - How read json with duplicated columns via spark?

Question

I am reading json by spark there is nothing special just:

spark.read.option('compression', 'gzip').option('dropFieldIfAllNull', True)
.json(source_final)

But it fails with : Found duplicate column(s) in the data schema. There are no joins just 2 JSON lines in 500 Mb file have duplicated fields and entire spark job failed. Any workarounds?

question from:https://stackoverflow.com/questions/66051974/how-read-json-with-duplicated-columns-via-spark

Answer 1

Waitting for answers