Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
532 views
in Technique[技术] by (71.8m points)

scala - spark shell column multiplication and updating same dataframe

I have a dataframe result

  +-----+--------+----------+-----+------------------+
  |count|currency|      date|value|         converted|
  +-----+--------+----------+-----+------------------+
  |    3|     USD|2021-01-14|    4|2.9311893333333336|
  |  102|     USD|2021-01-14|    3|               3.0|
  |  234|     USD|2021-01-14|    5| 3.663986666666667|
  |   68|     USD|2021-01-14|    6| 4.933771999999999|
  |   20|     USD|2021-01-14|    1|0.7327973333333334|
  |   28|     USD|2021-01-14|    5| 3.663986666666667|
  +-----+--------+----------+-----+------------------+

I want to multiply converted * count and store in another column in result

Desired Output

  +-----+--------+----------+-----+------------------+----------------+
  |count|currency|      date|value|         converted| convertedValue |
  +-----+--------+----------+-----+------------------+----------------+
  |    3|     USD|2021-01-14|    4|2.9311893333333336| 8.793568       |
  |  102|     USD|2021-01-14|    3|               3.0| 306            |
  |  234|     USD|2021-01-14|    5| 3.663986666666667| 857.37288      |
  |   68|     USD|2021-01-14|    6| 4.933771999999999| 335.496496     |
  |   20|     USD|2021-01-14|    1|0.7327973333333334| 14.6559466667  |
  |   28|     USD|2021-01-14|    5| 3.663986666666667| 102.591626667  |
  +-----+--------+----------+-----+------------------+----------------+

My Attempt and Error

    scala> result.withColumn("convertedValue", result["count"]*result["converted"]).show()
<console>:1: error: identifier expected but string literal found.
       result.withColumn("convertedValue", result["count"]*result["converted"]).show()
question from:https://stackoverflow.com/questions/65922917/spark-shell-column-multiplication-and-updating-same-dataframe

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Scala syntax for selecting column is different from Python. Try using parenthesis instead of square brackets:

result.withColumn("convertedValue", result("count")*result("converted")).show

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...