I'm trying to learn Spark following some hello-word level example such as below, using pyspark
. I got a "Method isBarrier([]) does not exist" error, full error included below the code.
from pyspark import SparkContext
if __name__ == '__main__':
sc = SparkContext('local[6]', 'pySpark_pyCharm')
rdd = sc.parallelize([1, 2, 3, 4, 5, 6, 7, 8])
rdd.collect()
rdd.count()
Although, when I start a pyspark session in command line directly and type in the same code, it works fine:
My setup:
- windows 10 Pro x64
- python 3.7.2
- spark 2.3.3 hadoop 2.7
- pyspark 2.4.0
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…