python - App Engine Deferred: Tracking Down Memory Leaks

Question

We have an App Engine application that writes many files of a relatively large size to Google Cloud Store. These files are CSVs that are dynamically created, so we use Python's StringIO.StringIO as a buffer and csv.writer as the interface for writing to that buffer.

In general, our process looks like this:

# imports as needed
# (gcs is the Google Cloud Store client)

buffer = StringIO.StringIO()
writer = csv.writer(buffer)

# ...
# write some rows
# ...

data = file_buffer.getdata()
filename = 'someFilename.csv'

try:
    with gcs.open(filename, content_type='text/csv', mode='w') as file_stream:
        file_stream.write(data)
        file_stream.close()

except Exception, e:
    # handle exception
finally:
    file_buffer.close()

As we understand it, the csv.writer does not need to be closed itself. Rather, only the buffer above and the file_stream need be closed.

---

We run the above process in a deferred, invoked by App Engine's task queue. Ultimately, we get the following error after a few invocations of our task:

Exceeded soft private memory limit of 128 MB with 142 MB after servicing 11 requests total

Clearly, then, there is a memory leak in our application. However, if the above code is correct (which we admit may not be the case), then our only other idea is that some large amount of memory is being held through the servicing of our requests (as the error message suggests).

Thus, we are wondering if some entities are kept by App Engine during the execution of a deferred. We should also note that our CSVs are ultimately written successfully, despite these error messages.

Answer 1

Waitting for answers