One way to achieve that is to use withColumn
method:
old_df = sqlContext.createDataFrame(sc.parallelize(
[(0, 1), (1, 3), (2, 5)]), ('col_1', 'col_2'))
new_df = old_df.withColumn('col_n', old_df.col_1 - old_df.col_2)
Alternatively you can use SQL on a registered table:
old_df.registerTempTable('old_df')
new_df = sqlContext.sql('SELECT *, col_1 - col_2 AS col_n FROM old_df')
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…