php - file_get_contents not working with utf8

Question

I'm trying to get Thai characters from a website. I've tried:

$rawChapter = file_get_contents("URL");
$rawChapter = mb_convert_encoding($rawChapter, 'UTF-8', mb_detect_encoding($rawChapter, 'UTF-8, ISO-8859-1', true));

When I do this then the characters come back like:

???o?1éòá????D¤?áà??ìàòéò?·??o?o

But if I take the source of the page I'm trying to load and save that into my own .htm file on my localhost as a utf8 file then it loads the Thai characters correctly. Only when I try to load it from the site directly it breaks.

How can I fix this? What could be the problem?

I've also tried adding this context:

$context = stream_context_create(array(
            'http' => array(
                'method' => 'POST',
                'header' => implode("
", array(
                    'Content-type: application/x-www-form-urlencoded',
                    'Accept-Language: en-us,en;q=0.5',
                    'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'
                ))
            )
        ));

I've tried adding it alone, I've tried adding it with the mb_convert_encoding()... I feel like I've tried all combinations of this stuff and no success.

Answer 1

Change your Accept-Charset to UTF-8 because ISO-8859-1 does not support Thai characters. If you are running your PHP script on a windows machine, you may also use the windows-874 charset, and you may also try adding this header :

Content-Language: th

But in most cases, UTF-8 will handle pretty much most characters or character sets without any other declaration.

** UPDATE **

Very strange, but this works for me.

$opts = array(
  'http'=>array(
    'method'=>"GET",
    'header'=> implode("
", array(
                   'Content-type: text/plain; charset=TIS-620'
                   //'Content-type: text/plain; charset=windows-874'  // same thing
                ))
  )
);

$context = stream_context_create($opts);

//$fp = fopen('http://thaipope.org/webbible/01_002.htm', 'rb', false, $context);
//$contents = stream_get_contents($fp);
//fclose($fp);
$contents = file_get_contents("http://thaipope.org/webbible/01_002.htm",false, $context);

header('Content-type: text/html; charset=TIS-620');
//header('Content-type: text/html; charset=windows-874');  // same thing

echo $contents;

Apparently, I was wrong for this one about UTF-8. See here for more details. Though you can still have an UTF-8 output :

$in_charset = 'TIS-620';   // == 'windows-874'
$out_charset = 'utf-8';

$opts = array(
  'http'=>array(
    'method'=>"GET",
    'header'=> implode("
", array(
                   'Content-type: text/plain; charset=' . $in_charset
                ))
  )
);

$context = stream_context_create($opts);

$contents = file_get_contents("http://thaipope.org/webbible/01_002.htm",false, $context);
if ($in_charset != $out_charset) {
    $contents = iconv($in_charset, $out_charset, $contents);
}

header('Content-type: text/html; charset=' . $out_charset);

echo $contents;   // output in UTF-8