I have a task which needs to be run on 'most' objects in my database once every some period of time (once a day, once a week, whatever). Basically this means that I have some query that looks like this running in it's own thread.
for model_instance in SomeModel.objects.all():
do_something(model_instance)
(Note that it's actually a filter() not all() but none-the-less I still end up selecting a very large set of objects.)
The problem I'm running into is that after running for a while the thread is killed by my hosting provider because I'm using too much memory. I'm assuming all this memory use is happening because even though the QuerySet
object returned by my query initially has a very small memory footprint it ends up growing as the QuerySet
object caches each model_instance
as I iterate through them.
My question is, "what is the best way to iterate through almost every SomeModel
in my database in a memory efficient way?" or perhaps my question is "how do I 'un-cache' model instances from a django queryset?"
EDIT: I'm actually using the results of the queryset to build a series of new objects. As such, I don't end up updating the queried-for objects at all.
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…