html - Nokogiri and Xpath: find all text between two tags

Question

I'm not sure if it's a matter of syntax or differences in versions but I can't seem to figure this out. I want to take data that is inside a (non-closing) td from the h2 tag to the h3 tag. Here is what the HTML would look like.

<td valign="top" width="350">
    <br><h2>NameIWant</h2><br>
    <br>Town<br>

    PhoneNumber<br>
    <a href="mailto:[email protected]" class="links">[email protected]</a>
    <br>
    <a href="http://websiteIwant.com" class="links">websiteIwant.com</a>
    <br><br>    
    <br><img src="images/spacer.gif"/><br>

    <h3><b>I want to stop before this!</b></h3>
    Lorem Ipsum Yadda Yadda<br>
    <img src="images/spacer.gif" border="0" width="20" height="11" alt=""/><br>
    <td width="25">
        <img src="images/spacer.gif" border="0" width="20" height="8" alt=""/>
        <td valign="top" width="200"><img src="images/spacer.gif"/>
            <br>
            <br>

            <table cellspacing="0" cellpadding="0" border="0"/>205"&gt;<tr><td>
                <a href="http://dontneedthis.com">
                </a></td></tr><br>
            <table border="0" cellpadding="3" cellspacing="0" width="200">
            ...

The <td valign> doesn't close until the very bottom of the page which I think might be why I'm having problems.

My Ruby code looks like:

require 'open-uri'
require 'nokogiri'

@doc = Nokogiri::XML(open("http://www.url.com"))

content = @doc.css('//td[valign="top"] [width="350"]')

name = content.xpath('//h2').text
puts name // Returns NameIwant

townNumberLinks = content.search('//following::h2')
puts content // Returns <h2> NameIWant </h2>

As I understand it following syntax should "Selects everything in the document after the closing tag of the current node". If I try to use preceding like:

townNumberLinks = content.search('//preceding::h3')
// I get: <h3><b>I want to stop before this!</b></h3>

Hope I made it clear what I'm trying to do. Thanks!

Answer 1

It's not trivial. In the context of the nodes you selected (the td), to get everything between two elements, you need to perform an intersection of these two sets:

1. Set A: All the nodes preceding the first h3: //h3[1]/preceding::node()
2. Set B: All the nodes following the first h2: //h2[1]/following::node()

To perform an intersection, you can use the Kaysian method (after Michael Kay, who proposed it). The basic formula is:

A[count(.|B) = count(B)]

Applying it to your sets, as defined above, where A = //h3[1]/preceding::node(), and B = //h2[1]/following::node(), we have:

//h3[1]/preceding::node()[ count( . | //h2[1]/following::node()) = count(//h2[1]/following::node()) ]

which will select all elements and text nodes starting with the first <br> after the </h2> tag, to the whitespace text node after the last <br>, just before the next <h3> tag.

You can easily select just the text nodes between h2 and h3 replacing node() for text() in the expression. This one will return all text nodes (including whitespace and linebreaks) between the two headers:

//h3[1]/preceding::text()[ count( . | //h2[1]/following::text()) = count(//h2[1]/following::text()) ]