Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
701 views
in Technique[技术] by (71.8m points)

python - How to download scrapy images in to a dynamic folder?

I am able to download images through scrapy in to the "Full" folder but I need to make the name of the destination folder dynamic, like full/session_id, every time scrapy runs.

Is there any way to do this?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

I have not worked with the ImagesPipeline yet, but following the documentation, I'd override item_completed(results, items, info).

The original definition is:

def item_completed(self, results, item, info):
    if self.IMAGES_RESULT_FIELD in item.fields:
        item[self.IMAGES_RESULT_FIELD] = [x for ok, x in results if ok]
    return item

This should give you the result sets of the downloaded images including the path (seems there can be many images on one item).

If you now change this method in a subclass to move all files before setting the path, it should work as you want. You could set the target folder on your item in something like item['session_path']. You'd have to set this setting on each item, before returning/yielding your items from the spider.

The subclass with overriden method could then look like this:

import os, os.path
from scrapy.contrib.pipeline.images import ImagesPipeline

class SessionImagesPipeline(ImagesPipeline):
    def item_completed(self, results, item, info):
        # iterate over the local file paths of all downloaded images
        for result in [x for ok, x in results if ok]:
            path = result['path']
            # here we create the session-path where the files should be in the end
            # you'll have to change this path creation depending on your needs
            target_path = os.path.join((item['session_path'], os.basename(path)))

            # try to move the file and raise exception if not possible
            if not os.rename(path, target_path):
                raise ImageException("Could not move image to target folder")

            # here we'll write out the result with the new path,
            # if there is a result field on the item (just like the original code does)
            if self.IMAGES_RESULT_FIELD in item.fields:
                result['path'] = target_path
                item[self.IMAGES_RESULT_FIELD].append(result)

        return item

Even nicer would be to set the desired session path not in the item, but in the configuration during your scrapy run. For this, you would have to find out how to set config while the application is running and you'd have to override the constructor, I think.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...