Uploading large files with Python/Django

Question

I am wondering if there are any ramifications in uploading files that are roughly 4GB in size through a web app using Django/Python? I remember in the past streaming uploads using Java was the preferred method but does this still today or is it perfectly safe to do so with Django/Python?

Answer 1

Django will by default, put uploaded file data into memory if it is less than 2.5MB. Anything larger will be written to the server's /tmp directory and then copied across when the transfer completes. Many of Django's file upload settings can be customised, details are available in the documentation. You can also customise the file handling and you'll certainly want to do this.

Before we consider any technical constraints, uploading such large files with the browser will give the user a very poor experience. There is no feedback about how the transfer is going (although google chrome does display the upload status as a percentage) and no way to pause or resume transfers.

You are also likely to run into problems on the server. Apart from the extremely long time that each thread will be taken with dealing with the streamed data, you have the time it takes for the system to copy the resulting file from /tmp to its correct location.

Unless you are very confident that you can foresee any problem that the server might have with the uploads, I would suggest that this is a bad idea. It's pretty hard to find any information on this via google and there do seem to be a lot of hits that describe problems with large file uploads.

While Django is technically capable of receiving uploaded files this large, the very poor user experience and technical difficulties mean this may not be the best approach. Have you considered using dedicated software to handle the file transfer?