grafana - Prometheus Increase function alert

Question

So my goal is to set up alerting mechanism for network interface drops. I am using node_network_receive_drop_total which is a counter. My goal is to alert when drops occur meaning when count of drops go up. Lets say I have these values for drops over a period of time.

|node_network_receive_drop_total |difference
|0                               |0
|1                               |1
|5                               |4
|5                               |0  
|7                               |2

In this example, how can I use increase function and alert condition in grafana to alert when the difference is non-zero?

Answer 1

In the Query tab a graph with a name something like node network drops and query something like this: increase(node_network_receive_drop_total[5m]).

When there are no drops, this graph should have a flat line at 0. When there is a drop the graph will show a line at 1 for 5 mins after the drop.

In the Alert tab create an alert with condition of WHEN max() OF query(A, 15m, now) IS ABOVE 0.

As you will only have one value returned by the query you could use max(), min(), or max() (they will all return the same value).

The A in query should match the letter on your query in the Query tab. If you have more than one query displayed on the graph, you may need to change this.

15m and now look at data from the last 15m.