I am following this blog to get Change Capture Using AWS Database Migration Service and Hudi
Running spark on my MacBook pro. getting the following error while reading parquet file from AWS S3.
code:
spark.read.parquet("s3://<S3_bucket_name>/*").sort("updated_at").show
Error:
java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated
Spark version - 3.0.1
Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_265)
Any help would be appreciated.
question from:
https://stackoverflow.com/questions/66053474/error-while-reading-parquet-file-in-spark-shell-error-provider-org-apache-spark 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…