apache spark - PySpark Add new object in nested field if not exist

Question

Schema

 root
     |-- userId: string (nullable = true)
     |-- languageknowList: array (nullable = true)
     |    |-- element: struct (containsNull = false)
     |    |    |-- code: string (nullable = false)
     |    |    |-- description: string (nullable = false)
     |    |    |-- name: string (nullable = false)

The df has userId and languageknownList. Every user should know English, so English language is not present in languageknowList I have to add.

English
code: 10
description: English Language
name: English

Any one please help me.

question from:https://stackoverflow.com/questions/65600994/pyspark-add-new-object-in-nested-field-if-not-exist

Answer 1

You can create a new array of structs column and concat to the existing column:

import pyspark.sql.functions as F

english = F.struct(F.lit('10').alias('code'),
                   F.lit('English Language').alias('description'), 
                   F.lit('English').alias('name')
                  )

df2 = df.withColumn(
    'languageknowList',
    F.when(
        ~F.array_contains(F.col('languageknowList'), english),
        F.concat(
            F.col('languageknowList'),
            F.array(english)
        )
    ).otherwise(
        F.col('languageknowList')
    )
)