apache spark - How to convert Pyspark Row datetime.datetime dataframe to column name DateType with timestamp in DD-MM-YYYY

Question

I m working on NYK Green Taxi Data Jan 2017,I converted string column to Datetype now I can see only in row as alias , How to convert it into dataframe to Show() as table in Pickupdate column and DD-MM-YYYYFormat

-- lpep_pickup_datetime: string (nullable = true)

df2 = df.select(to_timestamp(df.lpep_pickup_datetime, 'yyyy-MM-dd HH:mm:ss').alias('pickup_datetime')).collect()

df2
[Row(pickup_datetime=datetime.datetime(2017, 1, 1, 0, 1, 15)), Row(pickup_datetime=datetime.datetime(2017, 1, 1, 0, 3, 34)), Row(pickup_datetime=datetime.datetime(2017, 1, 1, 0, 4, 2)), Row(pickup_datetime=datetime.datetime(2017, 1, 1, 0, 1, 40)), Row(pickup_datetime=datetime.datetime(2017, 1, 1, 0, 0, 51)), Row(pickup_datetime=datetime.datetime(2017, 1, 1, 0, 0, 28))

Output should be DD-MM-YYYY format Pickup_datetime Column like 1-1-2017 00:01:15

question from:https://stackoverflow.com/questions/65623153/how-to-convert-pyspark-row-datetime-datetime-dataframe-to-column-name-datetype-w

Answer 1

Don't do collect which will convert your dataframe into a list of Rows. Just do the select, which will return a new dataframe df2, and call show() on that new dataframe.

df2 = df.select(to_timestamp(df.lpep_pickup_datetime, 'yyyy-MM-dd HH:mm:ss').alias('pickup_datetime'))

df2.show()