scala - How to pass -D parameter or environment variable to Spark job?

Question

I want to change Typesafe config of a Spark job in dev/prod environment. It seems to me that the easiest way to accomplish this is to pass -Dconfig.resource=ENVNAME to the job. Then Typesafe config library will do the job for me.

Is there way to pass that option directly to the job? Or maybe there is better way to change job config at runtime?

EDIT:

• Nothing happens when I add --conf "spark.executor.extraJavaOptions=-Dconfig.resource=dev" option to spark-submit command.
• I got Error: Unrecognized option '-Dconfig.resource=dev'. when I pass -Dconfig.resource=dev to spark-submit command.

Answer 1

Change spark-submit command line adding three options:

• --files <location_to_your_app.conf>
• --conf 'spark.executor.extraJavaOptions=-Dconfig.resource=app'
• --conf 'spark.driver.extraJavaOptions=-Dconfig.resource=app'