scala - How to create a DataSet from RDD using Case Class with composed columns

Question

Hi guys i using Spark/Scala 2.4.4 and have an RDD[Map(String, String)] with Persons... the rows are like this:

rdd = Map(
(FirstName -> "Michael")
(SecondName -> "Jackson")
)

I created a case class like this:

case class Person(FirstName: String, SecondName: String) {
    val FullName = FirstName + SecondName
}

how can i convert this RDD to a Dataset[Person], and it's possible to generate FullName column when the class are instantiated or i need to run withcolumn?

Thanks!

question from:https://stackoverflow.com/questions/65903174/how-to-create-a-dataset-from-rdd-using-case-class-with-composed-columns

Answer 1

You need to expand the map into two columns and cast the rows as Person.

val rdd = sc.parallelize(Seq(Map(
    ("FirstName" -> "Michael"),
    ("SecondName" -> "Jackson")
)))

val df = rdd.toDF("person").select("person.FirstName", "person.SecondName").as[Person]

Then you can get the FullName using map, for example:

df.map(r => r.FullName).show
+--------------+
|         value|
+--------------+
|MichaelJackson|
+--------------+