see scrapy Request structure, to crawl such chain you'll have to use the callback parameter like the following:
class MySpider(BaseSpider):
...
# spider starts here
def parse(self, response):
...
# A, D, E are done in parallel, A -> B -> C are done serially
yield Request(url=<A url>,
...
callback=parseA)
yield Request(url=<D url>,
...
callback=parseD)
yield Request(url=<E url>,
...
callback=parseE)
def parseA(self, response):
...
yield Request(url=<B url>,
...
callback=parseB)
def parseB(self, response):
...
yield Request(url=<C url>,
...
callback=parseC)
def parseC(self, response):
...
def parseD(self, response):
...
def parseE(self, response):
...
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…