python - 如何计算下降点数？(How to count number of decreasing point?)

Question

I want to count how many large decreasing point before the last point.

(我想计算最后一点之前有多少个大的下降点。)

Let say I have this dataframe

(假设我有这个数据框)

df = pd.DataFrame([[('2000-09-29 22:00:00'), 78.48659257615778],
       [('2000-09-29 23:00:00'), 79.46896724006444],
       [('2000-09-30 00:00:00'), 79.74108445117001],
       [('2000-09-30 01:00:00'), 79.59727418822389],
       [('2000-09-30 02:00:00'), 79.73048396364723],
       [('2000-09-30 03:00:00'), 79.09930124357557],
       [('2000-09-30 04:00:00'), 78.28903340851555],
       [('2000-09-30 05:00:00'), 78.12229420958224],
       [('2000-09-30 06:00:00'), 78.89923606080113],
       [('2000-09-30 07:00:00'), 78.9620439478767],
       [('2000-09-30 08:00:00'), 78.92566676678388],
       [('2000-09-30 09:00:00'), 61.74819886499889],
       [('2000-09-30 10:00:00'), 35.235524848405]])
df.columns = ['DATE', 'V1']

plt.plot(pd.to_datetime(df['DATE']), df['V1'], marker='.', markersize=8)
plt.show()

I want to get the count how many point have large deviate before the last point.

(我想计算最后一点之前有多少个点有较大的偏差。)

In this case it will give one point that decreasing largely before the last point.

(在这种情况下，将给出一个点，该点在最后一点之前大大减少。)

  ask by lemon93 translate from so

Answer 1

Iteratively identify points which lie outside 3 standard deviations.

(反复标识超出3个标准偏差的点。)

You might need to play around with the value 3 depending on what your data looks like.

(您可能需要根据数据的外观来计算值3。)

The iteration allows to base the cut-off decision only on the constant points, excluding the points after the drop.

(迭代允许仅基于恒定点（不包括落下后的点）进行截止决策。)

Otherwise, the result would depend more strongly on the number and (relative) values of the points with large standard deviation.

(否则，结果将更强烈地依赖于具有大标准偏差的点的数量和（相对）值。)

# this will be a dataframe where we delete all points with large deviation
df_cleaned = df[['V1']].copy()

mask_keep = [0, 1]  # just an initialization which satisfies the while condition
while sum(mask_keep) != len(mask_keep):  # while there is still points with large dev
    # normalize by subtracting mean
    df_cleaned['V1'] -= df_cleaned.V1.mean()
    # get mask of points which are inside the required range 
    # (3 times standard deviation in this case)
    mask_keep = df_cleaned.V1.abs() < 3 * df_cleaned.V1.std()
    # keep these points in df_cleaned, drop the rest
    df_cleaned = df_cleaned.loc[mask_keep]

# number of points with large deviation is length difference 
# of the two dataframes minus 1 (last point)
print('N points with large dev: ', len(df) - len(df_cleaned) - 1)

Output:

(输出：)

N points with large dev:  1