python - IOError: request data read error

Question

I seem to be getting an IOError: request data read error quite a lot when i'm doing an Ajax upload. For example out of every 5 file uploads it errors out on atleast 3.

Other people seem to have had the same issue. Eg.

• Django upload failing on request data read error
• Django file upload failing occasionally

Some other observations:

• It's definitely not my internet connection or a browser issue. Seems to be happening on all browsers chrome/FF/opera.

• I'm running django 1.1.1 Apache/2.2.14 (Ubuntu) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_wsgi/2.8 Python/2.6.5 on Lucid.

• It is also not the file size. I can sometimes upload 1+ MB files but fail on 180 Kb files.

---

Traceback

Traceback (most recent call last):

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/core/handlers/base.py", line 98, in get_response
    response = middleware_method(request, e)

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/core/handlers/base.py", line 92, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/contrib/auth/decorators.py", line 78, in __call__
    return self.view_func(request, *args, **kwargs)

  File "/home/ubuntu/webapps/anonymous_app/app/do_work/views/__init__.py", line 391, in some_form_ajax_upload
    f = request.FILES.get('file_upload')

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 187, in _get_files
    self._load_post_and_files()

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 137, in _load_post_and_files
    self._post, self._files = self.parse_file_upload(self.META, self.environ['wsgi.input'])

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/__init__.py", line 124, in parse_file_upload
    return parser.parse()

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 133, in parse
    for item_type, meta_data, field_stream in Parser(stream, self._boundary):

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 606, in __iter__
    for sub_stream in boundarystream:

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 420, in next
    return LazyStream(BoundaryIter(self._stream, self._boundary))

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 446, in __init__
    unused_char = self._stream.read(1)

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 299, in read
    out = ''.join(parts())

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 292, in parts
    chunk = self.next()

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 314, in next
    output = self._producer.next()

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 375, in next
    data = self.flo.read(self.chunk_size)

  File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 405, in read
    return self._file.read(num_bytes)

IOError: request data read error


<WSGIRequest
GET:<QueryDict: {}>,
POST:<could not parse>,
COOKIES:{'__utma': '168279989.1688771210.1285773436.1285773436.1285773436.1',
 '__utmb': '168279989.20.10.1285773436',
 '__utmc': '168279989',
 '__utmz': '168279989.1285773436.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',
 'beta': 'True',
 'sessionid': 'b1ecf92f2bba13e1885d07803e10aa03',
 'timezone_offset': '-330'},
META:{'CONTENT_LENGTH': '188575',
 'CONTENT_TYPE': 'multipart/form-data; boundary=---------------------------57602381214905740261171925981',
 'DOCUMENT_ROOT': '/htdocs',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTPS': '1',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_COOKIE': 'beta=True; __utma=168279989.1688771210.1285773436.1285773436.1285773436.1; __utmb=168279989.20.10.1285773436; __utmc=168279989; __utmz=168279989.1285773436.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); sessionid=b1ecf92f2bba13e1885d07803e10aa03; timezone_offset=-330',
 'HTTP_HOST': 'xxxxxx.compute-1.amazonaws.com',
 'HTTP_KEEP_ALIVE': '115',
 'HTTP_REFERER': 'https://ec2-184-72-79-96.compute-1.amazonaws.com/do-my-somees/enter/some-documents/',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10',
 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin',
 'PATH_INFO': u'/do-my-somees/enter/some-documents/ajax-upload/Other-some-Document/',
 'PATH_TRANSLATED': '/home/ubuntu/webapps/anonymous_app/settings/apache/qa.wsgi.py/do-my-somees/enter/some-documents/ajax-upload/Other-some-Document/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '',
 'REMOTE_PORT': '15561',
 'REQUEST_METHOD': 'POST',
 'REQUEST_URI': '/do-my-somees/enter/some-documents/ajax-upload/Other-some-Document/',
 'SCRIPT_FILENAME': '/home/ubuntu/webapps/anonymous_app/settings/apache/qa.wsgi.py',
 'SCRIPT_NAME': u'',
 'SERVER_ADDR': '10.196.142.182',
 'SERVER_ADMIN': 'dev@anonymous_app.com',
 'SERVER_NAME': 'ec2-184-72-79-96.compute-1.amazonaws.com',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SIGNATURE': '<address>Apache/2.2.14 (Ubuntu) Server at ec2-184-72-79-96.compute-1.amazonaws.com Port 443</address>
',
 'SERVER_SOFTWARE': 'Apache/2.2.14 (Ubuntu)',
 'SSL_TLS_SNI': 'ec2-184-72-79-96.compute-1.amazonaws.com',
 'mod_wsgi.application_group': 'qa.anonymous_app.com|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '443',
 'mod_wsgi.process_group': '',
 'mod_wsgi.reload_mechanism': '0',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (2, 8),
 'wsgi.errors': <mod_wsgi.Log object at 0xb9456860>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xb936a968>,
 'wsgi.input': <mod_wsgi.Input object at 0xb9720e30>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

Answer 1

I get this exception, too. In the Apache error logfile I see this:

[Wed Aug 17 08:30:45 2011] [error] [client 10.114.48.206] (70014)End of file found: mod_wsgi (pid=9722): Unable to get bucket brigade for request., referer: https://egs-work/modwork/beleg/188074/edit/
[Wed Aug 17 08:30:45 2011] [error] [client 10.114.48.206] mod_wsgi (pid=3572): Exception occurred processing WSGI script '/home/modwork_egs_p/modwork_egs/apache/django_wsgi.py'.
[Wed Aug 17 08:30:45 2011] [error] [client 10.114.48.206] IOError: failed to write data

Versions:

apache2-prefork-2.2.15-3.7.x86_64
apache2-mod_wsgi-3.3-1.8.x86_64 WSGIDaemonProcess with threads=1
mod_ssl/2.2.15
Linux egs-work 2.6.34.8-0.2-default #1 SMP 2011-04-06 18:11:26 +0200 x86_64 x86_64 x86_64 GNU/Linux
openSUSE 11.3 (x86_64)

First I was confused, because the last line "failed to write data" does not fit to the django code "load post data". But I guess that django wants to write an error page to the client. But the client has canceled the tcp connection. And now http 500 page can't be written to the client.

The client disconnected after sending the request, and before getting the response:

• The user closed the browser or navigated to an other page.
• The user pressed the reload button.

I have seen this only with POST-Requests (not GET). If POST is used, the webserver does read at least twice: First to get the headers, the second to get the data. The second read fails.

It is easy to reproduce:

Insert some code which waits before the first access to request.POST happens (be sure, that no middleware accesses request.POST before time.sleep()):

def edit(request):
    import time
    time.sleep(3)
    #.....

Now do a big POST (e.g. file upload). I don't know the apache buffer size. But 5 MB should be enough. When the browser shows the hourglass, browse to an other page. The browser will cancel the request and the exception should be in the logfile.

This is my Middleware, since I don't want to get the above traceback in our logfiles:

class HandleExceptionMiddleware:

    def process_exception(self, request, exception):
        if isinstance(exception, IOError) and 'request data read error' in unicode(exception):
            logging.info('%s %s: %s: Request was canceled by the client.' % (
                    request.build_absolute_uri(), request.user, exception))
            return HttpResponseServerError()