I have given two dataframes below for you to test
df = pd.DataFrame({
'subject_id':[1,1,1,1,1,1,1,1,1,1,1],
'time_1' :['2173-04-03 12:35:00','2173-04-03 17:00:00','2173-04-03
20:00:00','2173-04-04 11:00:00','2173-04-04 11:30:00','2173-04-04
12:00:00','2173-04-05 16:00:00','2173-04-05 22:00:00','2173-04-06
04:00:00','2173-04-06 04:30:00','2173-04-06 06:30:00'],
'val' :[5,5,5,10,5,10,5,8,3,8,10]
})
df1 = pd.DataFrame({
'subject_id':[1,1,1,1,1,1,1,1,1,1,1],
'time_1' :['2173-04-03 12:35:00','2173-04-03 12:50:00','2173-04-03
12:59:00','2173-04-03 13:14:00','2173-04-03 13:37:00','2173-04-04
11:30:00','2173-04-05 16:00:00','2173-04-05 22:00:00','2173-04-06
04:00:00','2173-04-06 04:30:00','2173-04-06 08:00:00'],
'val' :[5,5,5,5,10,5,5,8,3,4,6]
})
what I would like to do is
1) Find all values (from val
column) which have been same for more than 1 hour
in each day for each subject_id
and get the minimum of it
Please note that values can also be captured at every 15 min duration
as well, so you might have to consider 5 records to see > 1 hr
condition). See sample screenshot below
2) If there are no values which were same for more than 1 hour
in a day, then just get the minimum of that day for that subject_id
The below screenshot for one subject will help you understand and the code I tried is given below
This is what I tried
df['time_1'] = pd.to_datetime(df['time_1'])
df['time_2'] = df['time_1'].shift(-1)
df['tdiff'] = (df['time_2'] - df['time_1']).dt.total_seconds() / 3600
df['reading_day'] = pd.DatetimeIndex(df['time_1']).day
# don't know how to apply if else condition here to check for 1 hr criteria
t1 = df.groupby(['subject_id','reading_start_day','tdiff])['val'].min()
As I have to apply this to million records, any elegant and efficient solution would be helpful
See Question&Answers more detail:
os