Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.2k views
in Technique[技术] by (71.8m points)

monitoring - How can I 'join' two metrics in a Prometheus query?

I am using the consul exporter to ingest the health and status of my services into Prometheus. I'd like to fire alerts when the status of services and nodes in Consul is critical and then use tags extracted from Consul when routing those alerts.

I understand from this discussion that service tags are likely to be exported as a separate metric, but I'm not sure how to join one series with another so I can leverage the tags with the health status.

For example, the following query:

max(consul_health_service_status{status="critical"}) by (service_name, status,node) == 1

could return:

{node="app-server-02",service_name="app-server",status="critical"} 1

but I'd also like 'env' from this series:

consul_service_tags{node="app-server-02",service_name="app-server",env="prod"} 1

to get joined along node and service_name to pass the following to the Alertmanager as a single series:

{node="app-server-02",service_name="app-server",status="critical",env="prod"} 1

I could then match 'env' in my routing.

Is there any way to do this? It doesn't look to me like any operations or functions give me the ability to group or join like this. As far as I can see, the tags would already need to be labels on the consul_health_service_status metric.

question from:https://stackoverflow.com/questions/44461801/how-can-i-join-two-metrics-in-a-prometheus-query

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):

(
  max(consul_health_service_status{status="critical"}) 
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)

The important part here is the operation + on(service_name,node) group_left(env):

  • the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
  • group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...