scala - Error while reading Parquet file in spark-shell Error: Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated

Question

I am following this blog to get Change Capture Using AWS Database Migration Service and Hudi

Running spark on my MacBook pro. getting the following error while reading parquet file from AWS S3. code:

spark.read.parquet("s3://<S3_bucket_name>/*").sort("updated_at").show

Error:

java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated

Spark version - 3.0.1

Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_265)

Any help would be appreciated.

question from:https://stackoverflow.com/questions/66053474/error-while-reading-parquet-file-in-spark-shell-error-provider-org-apache-spark

Answer 1

Waitting for answers