Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
213 views
in Technique[技术] by (71.8m points)

python - Download a full page with scrapy

I want to download the content a whole page using scrapy.

With selenium this is quite easy:

import os,sys
reload(sys)  
sys.setdefaultencoding('utf8')
from selenium import webdriver


url = 'https://es.wikipedia.org/wiki/Python'

driver = webdriver.Firefox()
driver.get(url)
content = driver.page_source
with open('source','w') as output:
    output.write(content)

But selenium is much slower than scrapy.

Is it an simple way to do in scrapy?

I want to save the code of each page in a different file text, not as a csv or json file. Also, if posible without creating a project, which seems a bit of overkill for such a simple task.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Code will download this page and save it in file download-a-full-page-with-scrapy.html

test_scr.py

import scrapy
class TestSpider(scrapy.Spider):
    name = "test"

    start_urls = [
        "http://stackoverflow.com/questions/38233614/download-a-full-page-with-scrapy",
    ]

    def parse(self, response):
        filename = response.url.split("/")[-1] + '.html'
        with open(filename, 'wb') as f:
            f.write(response.body)

run scrapy by this command

scrapy runspider test_scr.py

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...