I am trying to create a new column by adding two existing columns in my dataframe.
Original dataframe
╔══════╦══════╗
║ cola ║ colb ║
╠══════╬══════╣
║ 1 ║ 1 ║
║ null ║ 3 ║
║ 2 ║ null ║
║ 4 ║ 2 ║
╚══════╩══════╝
Expected output with derived column
╔══════╦══════╦══════╗
║ cola ║ colb ║ colc ║
╠══════╬══════╬══════╣
║ 1 ║ 1 ║ 2 ║
║ null ║ 3 ║ 3 ║
║ 2 ║ null ║ 2 ║
║ 4 ║ 2 ║ 6 ║
╚══════╩══════╩══════╝
When I use df = df.withColumn('colc',df.cola+df.colb), it doesn't add columns with null values.
The output I get is:
╔══════╦══════╦══════╗
║ cola ║ colb ║ colc ║
╠══════╬══════╬══════╣
║ 1 ║ 1 ║ 2 ║
║ null ║ 3 ║ null ║
║ 2 ║ null ║ null ║
║ 4 ║ 2 ║ 6 ║
╚══════╩══════╩══════╝
Is there any way to incorporate the null values into the calculation. Any help would be appreciated.
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…