python - What is the method to add new column in existing dataframe in pyspark

Question

I have added a new column to an existing datframe but it's not reflected in dataframe.

customerDf.withColumn("fullname",expr("concat(firstname,'|',lastname)"))

customerDf.show() # it's showing existing old df records without new columns.

we can see the results if we can assign the dataframe to another dataframe

test = customerDf.withColumn("fullname",expr("concat(firstname,'|',lastname)"))
test.show()

Is there any way to add a new column to an existing dataframe (without copy dataframe)? We will have one option (inplace=True in pandas). Do we have any similar function in pyspark?

question from:https://stackoverflow.com/questions/65896658/what-is-the-method-to-add-new-column-in-existing-dataframe-in-pyspark

Answer 1

Short answer: no there is no such thing in pyspark.

Spark DataFrames are immutable. This means, when you add a new column (or any other transformation) you're not changing the data frame, but creating a new one.

https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.withColumn:

Returns a new DataFrame by adding a column or replacing the existing column that has the same name.

In Python you can, however, re-assign the result to "same variable" :

customerDf = customerDf.withColumn("fullname",expr("concat(firstname,'|',lastname)"))