Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
254 views
in Technique[技术] by (71.8m points)

So many variables in logstash

I'm planning to convert to a new logstash config to remodel my indexing and i added an if/if else/else filter to logstash. Right now the cluster has around 85 active indices which roll daily. Almost half of it will be monthly, around 25-30 of it will be weekly and the rest will be inidexed daily. I tested the conf on a test environment and i didnt encounter much problems. My question is will i experience significant performance drop after i apply this to that much indices. As a result i'll have lots of "or" statements in the if/else filter. What downsides will that have? Will it cause the logs to experience latency or worse data loss? Logstash jvm is 4gb at the moment. The logstash filter looks like this but with the number of variables i mentioned above:

filter {
        if [tag] == "x" or [tag] == "y" {
        mutate { add_field => { "[@metadata][target_index]" => "%{tag}-%{+YYYY.MM.dd}" } }
      } else if [tag] == "z" or [tag] == "t" {
        mutate { add_field => { "[@metadata][target_index]" => "%{tag}-%{+YYYY.MM}" } }
      } else {
        mutate { add_field => { "[@metadata][target_index]" => "%{tag}-%{+xxxx.ww}" } }
      }
    }

The indexing might change to adjust the number of given variables. For example monthly indices will be much more than the other 2 so monthly filter might just become the else condition.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)
等待大神答复

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...