Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
185 views
in Technique[技术] by (71.8m points)

How to cast all columns of Spark dataset to string using Java

I have a dataset with so many columns and I want to cast all columns to the string using Java.

I tried below steps, I want to know if there is any better way to achieve this?

Dataset<Row> ds = ...;
JavaRDD<String[]> stringArrRDD = ds.javaRDD().map(row->{
          int length = row.length();
          String[] columns = new String[length];
          for(int i=0; i<length;i++){
              columns[i] = row.get(i) !=null? row.get(i).toString():"";
          }
       return  columns;});
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

You can iterate over columns:

for (String c: ds.columns()) {
    ds = ds.withColumn(c, ds.col(c).cast("string"));
}

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...