list - How to convert combined data set to dataframe in Python

Question

This is my input list:

input_data = ['ned','etainclub','codingart','codingman','ksc','imrahelk', 'newbijohn','coinfarmer165','ponzipanda','blockchainstudio','jisoooh0202', 'jamieinthedark','xinnong','bbooaae','onehand','osyvv','bluengel','jungjunghoon','duplicate','lucky2']

I have defined function predict and after calling this function on the data I get this result:

[{'ned': 'male'}, {'etainclub': 'male'}, {'codingart': 'male'}, {'codingman': 'male'}, {'ksc': 'male'}, {'imrahelk': 'male'}, {'newbijohn': 'male'}, {'coinfarmer165': 'male'}, {'ponzipanda': 'female'}, {'blockchainstudio': 'male'}, {'jisoooh0202': 'male'}, {'jamieinthedark': 'male'}, {'xinnong': 'male'}, {'bbooaae': 'female'}, {'onehand': 'male'}, {'osyvv': 'male'}, {'bluengel': 'male'}, {'jungjunghoon': 'male'}, {'duplicate': 'female'}, {'lucky2': 'male'}]

This table below is the target format I want to get from the above data:

name	gender
ned	male
etainclub	male
duplicate	female
lucky2	male

question from:https://stackoverflow.com/questions/65865003/how-to-convert-combined-data-set-to-dataframe-in-python

Answer 1

The predict_output is a list of dicts. You can first convert it to a list of lists by getting the items of each dict element then pass the result to spark.createDataFrame():

import itertools

data = itertools.chain(*[p.items() for p in predict_output])

df = spark.createDataFrame(data, ["name", "gender"])
df.show(5)

#+---------+------+
#|     name|gender|
#+---------+------+
#|      ned|  male|
#|etainclub|  male|
#|codingart|  male|
#|codingman|  male|
#|      ksc|  male|
#+---------+------+