It's a really deep topic with a bunch of solutions. But if you want to apply the logic you've defined in your post you can use scrapy Downloader Middlewares.
Something like:
class CaptchaMiddleware(object):
max_retries = 5
def process_response(request, response, spider):
if not request.meta.get('solve_captcha', False):
return response # only solve requests that are marked with meta key
catpcha = find_catpcha(response)
if not captcha: # it might not have captcha at all!
return response
solved = solve_captcha(captcha)
if solved:
response.meta['catpcha'] = captcha
response.meta['solved_catpcha'] = solved
return response
else:
# retry page for new captcha
# prevent endless loop
if request.meta.get('catpcha_retries', 0) == max_retries:
logging.warning('max retries for captcha reached for {}'.format(request.url))
raise IgnoreRequest
request.meta['dont_filter'] = True
request.meta['captcha_retries'] = request.meta.get('captcha_retries', 0) + 1
return request
This example will intercept every response and try to solve the captcha. If failed it will retry the page for new captcha, if successful it will add some meta keys to response with solved captcha values.
In your spider you would use it like this:
class MySpider(scrapy.Spider):
def parse(self, response):
url = ''# url that requires captcha
yield Request(url, callback=self.parse_captchad, meta={'solve_captcha': True},
errback=self.parse_fail)
def parse_captchad(self, response):
solved = response['solved']
# do stuff
def parse_fail(self, response):
# failed to retrieve captcha in 5 tries :(
# do stuff
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…