php - unable to scrape content from a website

Question

I am trying to scrap some content from a website but the code below is not working(not showing any output). here is the code

$url="some url";
$otherHeaders="";   //here i am using some other headers like content-type,userAgent,etc
some curl to get the webpage
...
..
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
$content=curl_exec($ch);curl_close($ch);

$page=new DOMDocument();
$xpath=new DOMXPath($page); 
$content=getXHTML($content);  //this is a tidy function to convert bad html to xhtml 
$page->loadHTML($content);    // its okay till here when i echo $page->saveHTML the page is displayed

$path1="//body/table[4]/tbody/tr[3]/td[4]";
$path2="//body/table[4]/tbody/tr[1]/td[4]";

$item1=$xpath->query($path1);
$item2=$xpath->query($path2);

echo $item1->length;      //this shows zero 
echo $item2->length;      //this shows zero

foreach($item1 as $t)
echo $t->nodeValue;    //doesnt show anything
foreach($item2 as $p)
echo $p->nodeValue;    //doesnt show anything

i am sure there is something wrong with the above xpath code. the xpaths are correct. I have checked the above xpaths with FirePath (a firefox addon). I know i am missing something very silly here but i cant make out. Please help. I have checked similar code for scraping links from Wikipedia(definitely the xpaths are different) and it works nicely. So i dont understand why the above code does not work for the other URLs. I am cleaning the HTML content with Tidy so i dont there is a problem with xpath not geeting the HTML right? i have checked the length of the nodelist after $item1=$xpath->query($path1) which is 0 which means something is going wrong with $xpath->query because the xpaths are correct as i have checked with FirePath I have modified my code a bit as pointed out and used loadXML instead of loadHTML. but this gives me error as Entity 'nbsp' not defined in Entity so i used the libxml option LIBXML_NOENT to substitute entities but still the errors remain.

Answer 1

Yes, you are missing something very basic: It's XHTML, so you must register (and use!) the proper namespace before you can expect to get results.

$xpath->registerNamespace('x', 'http://www.w3.org/1999/xhtml');

$path1="//x:body/x:table[4]/x:tbody/x:tr[3]/x:td[4]";
$path2="//x:body/x:table[4]/x:tbody/x:tr[1]/x:td[4]";

$item1=$xpath->query($path1);
$item2=$xpath->query($path2);