python - Do I need to download Apache spark if I use pyspark in Pycharm?

Question

I'm new to pyspark and spark-nlp, I've been having quite a lot of issues with it when trying to run Python script in Pycharm within anaconda env, I happened see this page: https://github.com/JohnSnowLabs/spark-nlp/discussions/1022

It has some steps about how to correctly install Spark NLP on Windows 10, I'm not sure if I need to follow it:

• Step 2, it seems assume we have hadoop installed locally Download winutils and put it in C:hadoopin, but I don't have a folder name 'hadoop' do I need to download it?
• Step 4:Download Apache Spark 2.4.6 and extract it in C:spark I've installed pyspark 2.4.4 & spark-nlp 2.4.4 from conda/pip, do I still need to download Apache Spark??

I'm very confused now, because before seeing this article, I got an error py4j.protocol.Py4JJavaError: An error occurred while calling o314.load.: java.lang.ClassNotFoundException: com.johnsnowlabs.nlp.` in Pycharm (see this question: java.lang.ClassNotFoundException: com.johnsnowlabs.nlp.DocumentAssembler spark in Pycharm with conda env) and I wonder what's the right thing to do now...can someone help me please, thanks.

question from:https://stackoverflow.com/questions/66065584/do-i-need-to-download-apache-spark-if-i-use-pyspark-in-pycharm

Answer 1

Waitting for answers