I wanna set up scrapy cluster follow this link scrapy-cluster,Everything is ok before I run this command:
pip install -r requirements.txt
The requirements.txt looks like:
cffi==1.2.1
characteristic==14.3.0
ConcurrentLogHandler>=0.9.1
cryptography==0.9.1
...
I guess the above command means to install packages in requirements.txt.But I don't want it to specify the version,So I change it to this:
cat requirements.txt | while read line; do pip install ${line%%[>=]*} --user;done
When install cryptography,it gives me the error:
build/temp.linux-x86_64-2.7/_openssl.c:12:24:fatal error:pyconfig.h:No such file or directory
#include <pyconfig.h>
I don't know how to solved this , I have tried a lot of methods ,but failed. my system is centos 7, and the version of python is 2.7.5(default).
Besides, Is there any other scrapy frame which is appliable for a large number of urls . Thanks in advance
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…