apache spark - Dropping nested column of Dataframe with PySpark

Question

I'm trying to drop some nested columns in a Spark dataframe using pyspark. I found this for Scala that seems to be doing exactly what I want to but I'm not familiar with Scala and don't know how to write it in Python.

https://stackoverflow.com/a/39943812/5706548

I would really appreciate some help.

Thanks,

Answer 1

Example for pyspark:

def drop_col(df, struct_nm, delete_struct_child_col_nm):
    fields_to_keep = filter(lambda x:  x != delete_struct_child_col_nm, df.select("{}.*".format(struct_nm)).columns)
    fields_to_keep = list(map(lambda x:  "{}.{}".format(struct_nm, x), fields_to_keep))
    return df.withColumn(struct_nm, struct(fields_to_keep))