pyspark - The value of spark.network.timeout must be no less than the value of spark.executor.heartbeatInterval

Question

I am trying to increase the heartbeat interval parameter in pyspark configuration but keep getting this error. Is there any good combination of spark.executor.heartbeatInterval interval and spark.network.timeout that increase the timeout value for big jobs but also make me start spark session without this error?

IllegalArgumentException: u'requirement failed: The value of spark.network.timeout=120s must be no less than the value of spark.executor.heartbeatInterval=10000s.'

Thank you!

question from:https://stackoverflow.com/questions/65836076/the-value-of-spark-network-timeout-must-be-no-less-than-the-value-of-spark-execu

Answer 1

You have to increase the spark.network.timeout value too.

The documentation clearly states:

spark.executor.heartbeatInterval should be significantly less than spark.network.timeout

It's hard to interpret "significantly less" but by default spark.network.timeout is 12X greater (120s) than spark.executor.heartbeatInterval (10s).

Try:

--conf spark.network.timeout 100000 --conf spark.executor.heartbeatInterval 10000