I am trying to convert a scrapy script to a exe file.
The main.py file looks like this:
from scrapy.crawler import CrawlerProcess
from amazon.spiders.amazon_scraper import Spider
spider = Spider()
process = CrawlerProcess({
'FEED_FORMAT': 'csv',
'FEED_URI': 'data.csv',
'DOWNLOAD_DELAY': 3,
'RANDOMIZE_DOWNLOAD_DELAY': True,
'ROTATING_PROXY_LIST_PATH': 'proxies.txt',
'USER_AGENT_LIST': 'useragents.txt',
'DOWNLOADER_MIDDLEWARES' :
{
'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'random_useragent.RandomUserAgentMiddleware': 400
}
})
process.crawl(spider)
process.start() # the script will block here until the crawling is finished
The scrapy script looks like any other. I am using pyinstaller.exe --onefile main.py
to convert it to an exe file. When I try to open the main.exe file inside dist folder it starts outputing errors:
FileNotFoundError: [Errno 2] No such file or directory: '...\scrapy\VERSION'
I can fix it by creating a scrapy folder inside the dist folder and uploading a VERSION file from lib/site-packages/scrapy.
After that, many other errors occur but I can fix them by uploading some scrapy libraries.
In the end it starts outputing error:
ModuleNotFoundError: No module named 'email.mime'
I don`t even know what does it mean. I have never seen it.
I am using:
Python 3.6.5
Scrapy 1.5.0
pyinstaller 3.3.1
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…