Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
565 views
in Technique[技术] by (71.8m points)

python - How do I properly use connection pools in redis?

It's not clear to me how connections pools work, and how to properly use them. I was hoping someone could elaborate. I've sketched out my use case below:

settings.py:

import redis

def get_redis_connection():
    return redis.StrictRedis(host='localhost', port=6379, db=0)

task1.py

import settings

connection = settings.get_redis_connection()

def do_something1():
    return connection.hgetall(...)

task2.py

import settings

connection = settings.get_redis_connection()

def do_something1():
    return connection.hgetall(...)

etc.


Basically I have a setting.py file that returns redis connections, and several different task files that get the redis connections, and then run operations. So each task file has its own redis instance (which presumably is very expensive). What's the best way of optimizing this process. Is it possible to use connection pools for this example? Is there a more efficient way of setting up this pattern?

For our system, we have over a dozen task files following this same pattern, and I've noticed our requests slowing down.

Thanks

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Redis-py provides a connection pool for you from which you can retrieve a connection. Connection pools create a set of connections which you can use as needed (and when done - the connection is returned to the connection pool for further reuse). Trying to create connections on the fly without discarding them (i.e. not using a pool or not using the pool correctly) will leave you with way too many connections to redis (until you hit the connection limit).

You could choose to setup the connection pool in the init method and make the pool global (you can look at other options if uncomfortable with global).

redis_pool = None

def init():
    global redis_pool
    print("PID %d: initializing redis pool..." % os.getpid())
    redis_pool = redis.ConnectionPool(host='10.0.0.1', port=6379, db=0)

You can then retrieve the connection from a pool like this:

redis_conn = redis.Redis(connection_pool=redis_pool)

Also, I am assuming you are using hiredis along with redis-py as it should improve performance in certain cases. Have you also checked the number of connections open to the redis server with your existing setup as it most likely is quite high? You can use the INFO commmand to get that information:

redis-cli info

Check for the Clients section in which you will see the "connected_clients" field that will tell you how many connections you have open to the redis server at that instant.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...