I have not worked with the ImagesPipeline
yet, but following the documentation, I'd override item_completed(results, items, info)
.
The original definition is:
def item_completed(self, results, item, info):
if self.IMAGES_RESULT_FIELD in item.fields:
item[self.IMAGES_RESULT_FIELD] = [x for ok, x in results if ok]
return item
This should give you the result sets of the downloaded images including the path (seems there can be many images on one item).
If you now change this method in a subclass to move all files before setting the path, it should work as you want. You could set the target folder on your item in something like item['session_path']
. You'd have to set this setting on each item, before returning/yielding your items from the spider.
The subclass with overriden method could then look like this:
import os, os.path
from scrapy.contrib.pipeline.images import ImagesPipeline
class SessionImagesPipeline(ImagesPipeline):
def item_completed(self, results, item, info):
# iterate over the local file paths of all downloaded images
for result in [x for ok, x in results if ok]:
path = result['path']
# here we create the session-path where the files should be in the end
# you'll have to change this path creation depending on your needs
target_path = os.path.join((item['session_path'], os.basename(path)))
# try to move the file and raise exception if not possible
if not os.rename(path, target_path):
raise ImageException("Could not move image to target folder")
# here we'll write out the result with the new path,
# if there is a result field on the item (just like the original code does)
if self.IMAGES_RESULT_FIELD in item.fields:
result['path'] = target_path
item[self.IMAGES_RESULT_FIELD].append(result)
return item
Even nicer would be to set the desired session path not in the item
, but in the configuration during your scrapy run. For this, you would have to find out how to set config while the application is running and you'd have to override the constructor, I think.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…