java - Arabic text display issues

Question

Volley response is not showing the Arabic character ? Instead of this character I'm getting a diamond question mark ? . All the other characters are showing properly, I don't know the what is happening with this character alone.Is it the problem of volley web service? Any help appreciated.

    final RequestQueue requestQueue = Volley.newRequestQueue(Register.this);
    String url = Config.url + "validateID";
    StringRequest stringRequest = new StringRequest(Request.Method.POST, url, new   Response.Listener<String>() {
        @Override
        public void onResponse(String response) {
            response = response.trim();
            if (response != null) {
                try {
                    response = new String(response.getBytes(), "UTF-8");
                    response = Html.fromHtml(response).toString();
                    response = fixEncodingUnicode(response);
                    System.out.println("@@@@@@utf@@@@" + response);

                    try {
                        JSONObject jsonObject = new JSONObject(response);
                        String resp_code = jsonObject.getString("responseCode");
                        String status = jsonObject.getString("status");
                        if (resp_code.equals("200") && status.equals("ACTIVE")) {
                            ed_full_name.setText(jsonObject.getString("name"));
                            ed_full_name_arabic.setText(jsonObject.getString("namearabic"));
                        }
                     } catch (JSONException e) {
                        e.printStackTrace();
                     }
                } catch (UnsupportedEncodingException e) {
                    e.printStackTrace();
                }
            }
        }
    }, new Response.ErrorListener() {
        @Override
        public void onErrorResponse(VolleyError error) {
            progressDialog.dismiss();
    }


public static String fixEncodingUnicode(String response) {
    String str = "";
    try {
        str = new String(response.getBytes("windows-1254"), "UTF-8");
    } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
    }
    String decodedStr = Html.fromHtml(str).toString();
    return decodedStr;
}

question from:https://stackoverflow.com/questions/65661181/arabic-text-display-issues

Answer 1

I am afraid the patching code from a wrong encoding to a correct encoding cannot guarantee all characters survive the process. The main principle in java is that String always holds Unicode text; so there always is a conversion to bytes representing text in some encoding.

• response = new String(response.getBytes(), "UTF-8");

  This is wrong. getBytes() without charset uses the default charset from the platform which runs the current application. So it has a different effect on your development Windows PC and the production Linux server. Any effect is totally misleading.

• response = Html.fromHtml(response).toString();

  This encodes HTML entities. In a request a sign then the <form> is missing an accept-encoding="UTF-8". Part of the request headers. Then the browser sends non-Latin as HTML entities. Here it might be a communication failure between layers, where the request part is missing a UTF-8 accepting header.

• response = fixEncodingUnicode(response); or str = new String(response.getBytes("windows-1254"), "UTF-8");

  Unneeded as String in java already is in Unicode. It would introduce a diamond whenever a Unicode symbol was not translatable in Windows-1254.

So all seems wrong. The error seems to be made earlier on. Correct the requests, as otherwise a correct request might give wrong results. Go for UTF-8 rather than Windows-1254.

You can dump, log the bytes if the input parameter response, with something like:

Arrays.toString(response.codePoints().toArray())

(A hexadecimal format would be more readable.)