How to convert DataFrame columns from struct<value:double> to struct<values:array<double>> in pyspark?

Question

I have a DataFrame with this structure:

root
 |-- features: struct (nullable = true)
 |    |-- value: double (nullable = true)

and I wanna convert value with double type to "values with array" type. How can I do that?

question from:https://stackoverflow.com/questions/65844357/how-to-convert-dataframe-columns-from-structvaluedouble-to-structvaluesarra

Answer 1

You can specify the conversion explicitly using struct and array:

import pyspark.sql.functions as F

df.printSchema()
#root
# |-- features: struct (nullable = false)
# |    |-- value: double (nullable = false)

df2 = df.withColumn(
    'features',
    F.struct(
        F.array(F.col('features')['value']).alias('values')
    )
)

df2.printSchema()
#root
# |-- features: struct (nullable = false)
# |    |-- values: array (nullable = false)
# |    |    |-- element: double (containsNull = false)