This is an indicator of a Spark version mismatch. Before Spark 2.3 show
method took only two arguments:
def show(self, n=20, truncate=True):
since 2.3 it takes three arguments:
def show(self, n=20, truncate=True, vertical=False):
In your case Python client seems to invoke the latter one, while the JVM backend uses the older version.
Since SparkContext
initialization undergone significant changes in 2.4, which would cause failure on SparkContext.__init__
, you're likely using:
- 2.3.x Python library.
- 2.2.x JARs.
You can confirm that by checking versions directly from your session, Python:
sc.version
vs. JVM:
sc._jsc.version()
Problems like this, are usually a result of misconfigured PYTHONPATH
(either directly, or by using pip
installed PySpark
on top per-existing Spark binaries) or SPARK_HOME
.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…