I'm starting with scrapy, and I have first real problem. It's downloading pictures. So this is my spider.
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from example.items import ProductItem
from scrapy.utils.response import get_base_url
import re
class ProductSpider(CrawlSpider):
name = "product"
allowed_domains = ["domain.com"]
start_urls = [
"http://www.domain.com/category/supplies/accessories.do"
]
def parse(self, response):
hxs = HtmlXPathSelector(response)
items = []
sites = hxs.select('//td[@class="thumbtext"]')
number = 0
for site in sites:
item = ProductItem()
xpath = '//div[@class="thumb"]/img/@src'
item['image_urls'] = site.select(xpath).extract()[number]
item['image_urls'] = 'http://www.domain.com' + item['image_urls']
items.append(item)
number = number + 1
return items
When I quote ITEM_PIPELINES
and IMAGES_STORE
in settings.py
this way I get the proper URL for picture I want to download (copy pasted it into browser for check).
But when I unquote those i get following error:
raise ValueError('Missing scheme in request url: %s' % self._url')
exceptions.ValueError: Missing scheme in request url:h
and I can't download my pictures.
I've searched for the whole day and didn't find anything helpful.
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…