io - Reading a web page in Java IOException Premature EOF

Question

I am frequently getting a 'Premature EOF' Exception when reading a web page.

The following is the StackTrace

java.io.IOException: Premature EOF
    at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:556)
    at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:600)
    at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:687)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2968)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:154)
    at java.io.BufferedReader.readLine(BufferedReader.java:317)
    at java.io.BufferedReader.readLine(BufferedReader.java:382)
    at Utilities.getPage(Utilities.java:24)  while ((line = rd.readLine()) != null) {
    at TalkPage.<init>(TalkPage.java:15)
    at Updater.run(Updater.java:65)

Following is the getPage() method

public static String getPage(String urlString) throws Exception {
    URL url = new URL(urlString);
    URLConnection conn = url.openConnection();
    BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
    StringBuffer sb = new StringBuffer();
    String line;
    while ((line = rd.readLine()) != null) {  // LINE 24
        sb.append(line);
    }
    return sb.toString();
}

What is a permature EOFException and why is it occuring in this particular case and how can it be avoided?

Some other information: The size of the page being read is around 20 KB and I'm reading many such pages in my program ( around 20 000 )

Answer 1

This may be because you are reading the content line by line and for the last line the file may be missing a return, to signal the end of line. Replace your while with this:

int BUFFER_SIZE=1024;
char[] buffer = new char[BUFFER_SIZE]; // or some other size, 
int charsRead = 0;
while ( (charsRead  = rd.read(buffer, 0, BUFFER_SIZE)) != -1) {
  sb.append(buffer, 0, charsRead);
}