Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
608 views
in Technique[技术] by (71.8m points)

python - storing data into redis through cron job

I want to store data into redis from pandas through a cron job every 15 minute and below is my code:-

I am taking data into pandas every 15 minutes with below code and sending it to the redis dictionary mydict2 through a cron job.

import sys
import pickle
import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)

test_dict1 = results_df.set_index('user')['ua'].T.to_dict()

p_mydict = pickle.dumps(test_dict1)
r.set('mydict2', p_mydict)

I am getting the same output again and again in the key mydict2. Basically i want to store user ids for the whole month and at the end of the month i want the unique count of that.

Also i am using set method, what could be the best method assuming i am having a very large amount of data.

Can someone help me out here.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Replace below

p_mydict = pickle.dumps(test_dict1)
r.set('mydict2', p_mydict)

with

    for k, v in test_dict1.items():
        r.hmset(k, {"ua" : v})
    print("Done adding stuff")

and each key in your dictionary will be a key in Redis.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

1.4m articles

1.4m replys

5 comments

57.0k users

...