Write Spark data frame to multiple s3 buckets

Question

I want to save dataframe rows in the different S3 buckets. Let's assume all buckets exist. I have a simple dataframe:

tenantId	charge
tenant1	10
tenant2	20

question from:https://stackoverflow.com/questions/65850046/write-spark-data-frame-to-multiple-s3-buckets

Answer 1

Please use partitionBy clause as -

          df.write.
          partitionBy("tenantId").
          parquet("the root path")

It will create distinct folder by names "tenant1" and "tenant2" and put respective rows inside.