grafana - How to filter prometheus query results based on the maximum value in a time range?

Question

I am monitoring a series of CPU load data, which is aggregated. Similar to:

avg(load_data{}) by (service_name)

Now, I just want to filter out the results that meet the following conditions: In the past n days, the maximum value of one service is greater than 90%.

My purpose is to focus on the service who had a high CPU usage.

How can I do that? I tried max_over_time() but not work.

Answer 1

You need to use subquery, like this:

max_over_time(avg by (service_name) (load_data)[1d:])

More info in the Prometheus documentation here.