I'm scraping from a UTF-8 site, using Goutte, which internally uses Guzzle. The site declares a meta tag of UTF-8, thus:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
However, the content type header is thus:
Content-Type: text/html
and not:
Content-Type: text/html; charset=utf-8
Thus, when I scrape, Goutte does not spot that it is UTF-8, and grabs data incorrectly. The remote site is not under my control, so I can't fix the problem there! Here's a set of scripts to replicate the problem. First, the scraper:
<?php
require_once realpath(__DIR__ . '/..') . '/vendor/goutte/goutte.phar';
$url = 'http://crawler-tests.local/utf-8.php';
use GoutteClient;
$client = new Client();
$crawler = $client->request('get', $url);
$text = $crawler->text();
echo 'Whole page: ' . $text . "
";
Now a test page to be placed on a web server:
<?php
// Correct
#header('Content-Type: text/html; charset=utf-8');
// Incorrect
header('Content-Type: text/html');
?>
<!DOCTYPE html>
<html>
<head>
<title>UTF-8 test</title>
<meta charset="utf-8" />
</head>
<body>
<p>When the Content-Header header is incomplete, the pound sign breaks:
￡15,216</p>
</body>
</html>
Here's the output of the Goutte test:
Whole page: UTF-8 test
When the Content-Header header is incomplete, the pound sign breaks: ?￡15,216
As you can see from the comments in the last script, properly declaring the character set in the header fixes things. I've hunted around in Goutte to see if there is anything that looks like it would force the character set, but to no avail. Any ideas?
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
