html - how can i extract special kind of table from website in perl?

Question

I am trying to fetch all tables from the website http://finance.yahoo.com/etf/lists/?bypass=true&mod_id=mediaquotesetf&tab=tab1&scol=imkt&stype=desc&rcnt=50&page=1, using Perl module HTML::TableExtract, but I can't get the desired table; instead I get the first two tables only, which are useless to me.

Here is my code:

#!/usr/bin/perl
#!perl -w
use DBI;
use strict;
use WWW::Mechanize;
use HTML::TableExtract;
my $mech= WWW::Mechanize->new();
my $url= 'http://finance.yahoo.com/etf/lists/?bypass=true&mod_id=mediaquotesetf&tab=tab1&scol=imkt&stype=desc&rcnt=50&page=1';
$mech -> get($url);
chomp(my $script = $mech -> content);
my $table=new HTML::TableExtract();
$table->parse($script);

foreach my $ts($table->tables){

print "Table (", join(',', $ts->coords), "):
";

foreach my $row ($ts->rows){
    print join(',', @$row), "
";
}
}

output:

Table (0,0):
  ,Search FinanceSearch Web
Table (0,1):

                              Quotes you view appear here for quick access.

like this I only get the first two tables instead of all of them.

Answer 1

The third table is generated dynamically using JavaScript. WWW::Mechanize doesn't support JavaScript, and you will need to use WWW::Mechanize::Firefox instead

Note that this will require you to install a Firefox web browser, and its mozrepl plugin, as well as the MozRepl Perl module

use strict;
use warnings 'all';
use feature 'say';
use open qw/ :std :encoding(UTF-8) /;

use WWW::Mechanize::Firefox;
use HTML::TableExtract;

use constant URL => 'http://finance.yahoo.com/etf/lists/?bypass=true&mod_id=mediaquotesetf&tab=tab1&scol=imkt&stype=desc&rcnt=50&page=1';

my $mech = WWW::Mechanize::Firefox->new;
$mech->autoclose_tab(0);
$mech->get(URL);

my $te = HTML::TableExtract->new(depth => 0, count => 2);
$te->parse( $mech->content );

for my $row ( $te->rows ) {
    local $" = ',';
    print "@$row
";
}

output

ETF Name,Ticker,Category,Fund Family,Intraday Return,3-MO Return,YTD Return,1-YR Return,3-YR Return,5-YR Return
UBS ETRACS ISE Exclusively Hmbldrs ETN,HOMX,Consumer Cyclical,UBS Group AG,+13.43%,-6.74%,-6.74%,-19.54%,0.0%,0.0%
VelocityShares 3x Long Natural Gas ETN,UGAZ,Trading-Leveraged Commodities,Credit Suisse AG,+9.33%,-60.15%,-60.15%,-91.36%,-81.58%,0.0%
ProShares Ultra Bloomberg Natural Gas,BOIL,Trading-Leveraged Commodities,ProShares,+5.96%,-41.13%,-41.13%,-76.12%,-62.1%,0.0%
Direxion Daily Brazil Bull 3X ETF,BRZU,Trading-Leveraged Equity,Direxion Funds,+4.24%,+66.64%,+66.64%,-61.7%,0.0%,0.0%
DB Commodity Double Long ETN,DYY,Trading-Leveraged Commodities,Deutsche Bank AG,+4.16%,-25.87%,-25.87%,-41.98%,-34.91%,-32.8%
Deutsche X-trackers MSCI EMktsHiDvYdHgEq,HDEE,Diversified Emerging Mkts,Deutsche Asset Management,+3.73%,-2.15%,-2.15%,0.0%,0.0%,0.0%
DB Agriculture Double Long ETN,DAG,Trading-Leveraged Commodities,Deutsche Bank AG,+3.57%,+3.12%,+3.12%,-19.14%,-29.52%,-25.49%
United States Natural Gas,UNG,Commodities Energy,United States Commodity Funds LLC,+3.15%,-23.18%,-23.18%,-49.7%,-32.73%,-32.09%
Direxion Daily Jr Gld Mnrs Bear 3X ETF,JDST,Trading-Inverse Equity,Direxion Funds,+3.03%,-80.83%,-80.83%,-88.33%,0.0%,0.0%
Direxion Daily S&P Biotech Bull 3X ETF,LABU,Trading-Leveraged Equity,Direxion Funds,+2.97%,-67.51%,-67.51%,0.0%,0.0%,0.0%
VelocityShares 3x Inverse Silver ETN,DSLV,Trading-Inverse Commodities,Credit Suisse AG,+2.88%,-36.11%,-36.11%,-15.03%,+13.64%,0.0%
ProShares Ultra MSCI Brazil Capped,UBR,Trading-Leveraged Equity,ProShares,+2.85%,+51.79%,+51.79%,-37.19%,-42.04%,-38.48%
Direxion Daily India Bull 3X ETF,INDL,Trading-Leveraged Equity,Direxion Funds,+2.71%,-9.82%,-9.82%,-48.84%,-13.11%,-22.69%
Direxion Daily Real Estate Bull 3X ETF,DRN,Trading-Leveraged Equity,Direxion Funds,+2.66%,+15.65%,+15.65%,-0.71%,+21.8%,+22.37%
iShares US Telecommunications,IYZ,Communications,iShares,+2.63%,+7.43%,+7.43%,+3.79%,+10.74%,+7.88%
ProShares Ultra Semiconductors,USD,Trading-Leveraged Equity,ProShares,+2.58%,-4.09%,-4.09%,-6.65%,+31.6%,+15.12%
Direxion Daily Pharmctcl&Medcl Bl 2X ETF,PILL,Trading-Leveraged Equity,Direxion Funds,+2.57%,-27.81%,-27.81%,0.0%,0.0%,0.0%
IQ Hedge Event-Driven Tracker ETF,QED,Market Neutral,IndexIQ,+2.54%,+1.51%,+1.51%,-1.64%,0.0%,0.0%
Direxion Daily Regional Bnks Bull 3X ETF,DPST,Trading-Leveraged Equity,Direxion Funds,+2.51%,-20.73%,-20.73%,0.0%,0.0%,0.0%
VelocityShares 3x Inverse Gold ETN,DGLD,Trading-Inverse Commodities,Credit Suisse AG,+2.44%,-40.19%,-40.19%,-24.19%,+6.6%,0.0%
Direxion Daily South Korea Bull 3X ETF,KORU,Trading-Leveraged Equity,Direxion Funds,+2.43%,+12.48%,+12.48%,-30.37%,0.0%,0.0%
VelocityShares Daily Inverse VIX ST ETN,XIV,Volatility,Credit Suisse AG,+2.43%,+0.31%,+0.31%,-25.29%,+3.55%,+13.26%
ProShares Ultra S&P Regional Banking,KRU,Trading-Leveraged Equity,ProShares,+2.42%,-22.89%,-22.89%,-17.41%,+10.48%,+9.61%
Global X FTSE Andean 40 ETF,AND,Latin America Stock,Global X Funds,+2.38%,+12.8%,+12.8%,-15.28%,-19.56%,-11.0%
AccuShares Spot CBOE? VIX? ETC Down,VXDN,Volatility,AccuShares?,+2.32%,-12.72%,-12.72%,0.0%,0.0%,0.0%
ProShares Short S&P Regional Banking,KRS,Trading-Inverse Equity,ProShares,+2.3%,+7.46%,+7.46%,-0.5%,-12.4%,-14.52%
United States 12 Month Natural Gas,UNL,Commodities Energy,United States Commodity Funds LLC,+2.3%,-8.84%,-8.84%,-29.88%,-22.73%,-23.91%
ProShares Short VIX Short-Term Futures,SVXY,Volatility,ProShares,+2.28%,+0.16%,+0.16%,-25.73%,+3.53%,0.0%
SPDR? S&P Transportation ETF,XTN,Industrials,SPDR State Street Global Advisors,+2.24%,+7.3%,+7.3%,-12.73%,+12.63%,+12.34%
iShares MSCI UAE Capped,UAE,Miscellaneous Region,iShares,+2.22%,+5.31%,+5.31%,-4.98%,0.0%,0.0%
VelocityShares 3x Inverse Crude Oil ETN,DWTI,Trading-Inverse Commodities,Credit Suisse AG,+2.21%,-20.19%,-20.19%,+19.62%,+56.72%,0.0%
ProShares Ultra High Yield,UJB,Trading-Leveraged Debt,ProShares,+2.19%,+10.09%,+10.09%,-6.81%,+1.36%,0.0%
iPath? Bloomberg Livestock SubTR ETN,COW,Commodities Agriculture,Barclays Funds,+2.19%,+0.94%,+0.94%,-10.57%,-3.1%,-5.92%
iPath? Bloomberg Natural Gas SubTR ETN,GAZ,Commodities Energy,Barclays Funds,+2.19%,-31.94%,-31.94%,-59.17%,-44.91%,-49.91%
Direxion Daily Small Cap Bull 3X ETF,TNA,Trading-Leveraged Equity,Direxion Funds,+2.16%,-8.7%,-8.7%,-35.41%,+10.14%,+6.17%
ProShares Ultra Utilities,UPW,Trading-Leveraged Equity,ProShares,+2.16%,+30.83%,+30.83%,+28.99%,+22.85%,+24.31%
SPDR? Wells Fargo Preferred Stock ETF,PSK,Preferred Stock,SPDR State Street Global Advisors,+2.14%,+1.77%,+1.77%,+5.83%,+5.83%,+6.08%
ProShares UltraShort Silver,ZSL,Trading-Inverse Commodities,ProShares,+2.14%,-23.44%,-23.44%,-1.99%,+21.56%,-2.97%
ProShares UltraPro Russell2000,URTY,Trading-Leveraged Equity,ProShares,+2.14%,-8.53%,-8.53%,-34.81%,+10.76%,+7.06%
Direxion Daily Emrg Mkts Bull 3X ETF,EDC,Trading-Leveraged Equity,Direxion Funds,+2.11%,+13.64%,+13.64%,-44.79%,-25.84%,-28.1%
DB Agriculture Short ETN,ADZ,Trading-Inverse Commodities,Deutsche Bank AG,+2.1%,-11.59%,-11.59%,-5.04%,+11.94%,+7.62%
DB 3x Long 25+ Year Treasury Bond ETN,LBND,Trading-Leveraged Debt,Deutsche Bank AG,+2.08%,+22.67%,+22.67%,+2.15%,+11.31%,+24.21%
PureFunds ISE Cyber Security? ETF,HACK,Technology,Pure Funds,+2.02%,-7.45%,-7.45%,-14.3%,0.0%,0.0%
ProShares UltraPro MidCap400,UMDD,Trading-Leveraged Equity,ProShares,+1.95%,+6.04%,+6.04%,-19.63%,+20.32%,+16.2%
ProShares Ultra Telecommunications,LTL,Trading-Leveraged Equity,ProShares,+1.79%,+13.74%,+13.74%,+0.87%,+18.18%,+11.67%
Direxion Daily Hmbldrs&Supls Bull 3X ETF,NAIL,Trading-Leveraged Equity,Direxion Funds,+1.79%,-10.32%,-10.32%,0.0%,0.0%,0.0%
Teucrium Wheat ETF,WEAT,Commodities Agriculture,Teucrium,+1.78%,-1.54%,-1.54%,-17.67%,-21.21%,0.0%
Vanguard Telecommunication Services ETF,VOX,Communications,Vanguard,+1.76%,+11.11%,+11.11%,+11.82%,+11.64%,+9.99%
Direxion Daily Financial Bull 3X ETF,FAS,Trading-Leveraged Equity,Direxion Funds,+1.75%,-14.79%,-14.79%,-18.93%,+21.63%,+14.48%
US Global Jets ETF,JETS,Miscellaneous Sector,U.S. Global Investors,+1.75%,+1.77%,+1.77%,0.0%,0.0%,0.0%