apache spark - java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

Question

I am using Spark 1.5.0 and I have an issue while creating a dataframe from my rdd.

Here is the code:

case class MyC (myclass: MyClass) 

  val df = rdd.map {t => MyC(t)}.toDF("cust")
  df.show()

Here is the error message:

Exception in thread "main" java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

Any help with this will be greatly appreciated.

Answer 1

Spark uses reflection to infer dataframe schema, but cannot do so for arbitrary classes. I'm not sure if I can state an answer better than the Spark docs can, and there's a section dedicated to exactly this.

To be specific, the problem is that there are a limited number of types for which Spark can infer schema. Ints, Strings, Seqs/Arrays are all supported (as well as case classes containing elements of these types), but an arbitrary class MyClass is not.