python - How to convert a numeric type column into timestamp column?

Question

I have sample records as looks below and I want to convert the Time numeric column (Which represents milliseconds value) into the timestamp column.

Value   Time
0.00346 0
0.01584 0.002
0.00621 0.004
.....
.....
.....
0.00204 0.998
0.00617 1
0.00204 1.002

Let's take HH:mm:ss.SSS is the timestamp format and the output should be as below

Value   Time   timestamp
0.00346 0      00:00:00.000
0.01584 0.002  00:00:00.002
0.00621 0.004  00:00:00.004
.....          ........
.....          ........
.....          ........
0.00204 0.998  00:00:00.998
0.00617 1      00:00:01.000
0.00204 1.002  00:00:01.002

question from:https://stackoverflow.com/questions/65881353/how-to-convert-a-numeric-type-column-into-timestamp-column

Answer 1

You can cast the column to timestamp type and then use date_format to convert to the desired format:

import pyspark.sql.functions as F

df2 = df.withColumn('timestamp', F.date_format(F.col('Time').cast('timestamp'), 'HH:mm:ss.SSS'))

df2.show()
+-------+-----+------------+
|  Value| Time|   timestamp|
+-------+-----+------------+
|0.00346|  0.0|00:00:00.000|
|0.01584|0.002|00:00:00.002|
|0.00621|0.004|00:00:00.004|
|0.00204|0.998|00:00:00.998|
|0.00617|  1.0|00:00:01.000|
|0.00204|1.002|00:00:01.002|
+-------+-----+------------+