Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.1k views
in Technique[技术] by (71.8m points)

linux - How to limit memory usage within a python process

I run Python 2.7 on a Linux machine with 16GB Ram and 64 bit OS. A python script I wrote can load too much data into memory, which slows the machine down to the point where I cannot even kill the process any more.

While I can limit memory by calling:

ulimit -v 12000000

in my shell before running the script, I'd like to include a limiting option in the script itself. Everywhere I looked, the resource module is cited as having the same power as ulimit. But calling:

import resource
_, hard = resource.getrlimit(resource.RLIMIT_DATA)
resource.setrlimit(resource.RLIMIT_DATA, (12000, hard))

at the beginning of my script does absolutely nothing. Even setting the value as low as 12000 never crashed the process. I tried the same with RLIMIT_STACK, as well with the same result. Curiously, calling:

import subprocess
subprocess.call('ulimit -v 12000', shell=True)

does nothing as well.

What am I doing wrong? I couldn't find any actual usage examples online.


edit: For anyone who is curious, using subprocess.call doesn't work because it creates a (surprise, surprise!) new process, which is independent of the one the current python program runs in.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

resource.RLIMIT_VMEM is the resource corresponding to ulimit -v.

RLIMIT_DATA only affects brk/sbrk system calls while newer memory managers tend to use mmap instead.

The second thing to note is that ulimit/setrlimit only affects the current process and its future children.

Regarding the AttributeError: 'module' object has no attribute 'RLIMIT_VMEM' message: the resource module docs mention this possibility:

This module does not attempt to mask platform differences — symbols not defined for a platform will not be available from this module on that platform.

According to the bash ulimit source linked to above, it uses RLIMIT_AS if RLIMIT_VMEM is not defined.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...