Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
479 views
in Technique[技术] by (71.8m points)

python - How to extract raw html from a Scrapy selector?

I'm extracting js data using response.xpath('//*')re_first() and later converting it to python native data. The problem is extract/re methods don't seem to provide a way to not unquote html i.e.

original html:

{my_fields:['O'Connor Park'], }

extract output:

{my_fields:['O'Connor Park'], }

turning this output into json won't work.

What's the easiest way around it?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Short answer:

  • Scrapy/Parsel selectors' .re() and .re_first() methods replace HTML entities (except <, &)
  • instead, use .extract() or .extract_first() to get raw HTML (or raw JavaScript instructions) and use Python's re module on extracted string

Long answer:

Let's look at an example input and various ways of extracting Javascript data from HTML.

Sample HTML:

<html lang="en">
<body>
<div>
    <script type="text/javascript">
        var i = {a:['O&#39;Connor Park']}
    </script>
</div>
</body>
</html>

Using scrapy Selector, which is using the parsel library underneath, you have several ways of extracting the Javascript snippet:

>>> import scrapy
>>> t = """<html lang="en">
... <body>
... <div>
...     <script type="text/javascript">
...         var i = {a:['O&#39;Connor Park']}
...     </script>
...     
... </div>
... </body>
... </html>
... """
>>> selector = scrapy.Selector(text=t, type="html")
>>> 
>>> # extracting the <script> element as raw HTML
>>> selector.xpath('//div/script').extract_first()
u'<script type="text/javascript">
        var i = {a:['O&#39;Connor Park']}
    </script>'
>>> 
>>> # only getting the text node inside the <script> element
>>> selector.xpath('//div/script/text()').extract_first()
u"
        var i = {a:['O&#39;Connor Park']}
    "
>>> 

Now, Using .re (or .re_first) you get different result:

>>> # I'm using a very simple "catch-all" regex
>>> # you are probably using a regex to extract
>>> # that specific "O'Connor Park" string
>>> selector.xpath('//div/script/text()').re_first('.+')
u"        var i = {a:['O'Connor Park']}"
>>> 
>>> # .re() on the element itself, one needs to handle newlines
>>> selector.xpath('//div/script').re_first('.+')
u'<script type="text/javascript">'    # only first line extracted
>>> import re
>>> selector.xpath('//div/script').re_first(re.compile('.+', re.DOTALL))
u'<script type="text/javascript">
        var i = {a:['O'Connor Park']}
    </script>'
>>> 

The HTML entity &#39; has been replaced by an apostrophe. This is due to a w3lib.html.replace_entities() call in .re/re_first implementation (see parsel source code, in extract_regex function), which is not used when simply calling extract() or extract_first()


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

1.4m articles

1.4m replys

5 comments

57.0k users

...