How to set Kafka properties (in what property files) in Windows & monitor Kafka cluster health?

Question

I have Kafka setup on my Windows and have a local Kafka cluster of 2 brokers setup. I am trying to debug certain Kafka reliability issues when working with 2 brokers, one going down, both going down, etc., and have a couple of questions.

1. In what .properties file (just like we have server.properties) can I set the following Kafka related properties: a) broker.rack b) min.insync.replica c) unclean.leader.election.enable d) enable.auto.commit

2. Where to check for the Kafka consumer 'message' commit logs, when I'm trying to monitor my cluster health?

3. How do I monitor the 'out-of-sync' behavior of each of my brokers in the cluster? (need this just to see how erratically my brokers are going out of sync in the cluster, and if there is a serious flaw I need to unfold in the cluster configuration)

Answers to my questions, references, ways of debugging cluster health, and inputs are welcome!

question from:https://stackoverflow.com/questions/65850774/how-to-set-kafka-properties-in-what-property-files-in-windows-monitor-kafka

Answer 1

You can find all the properties for the server.properties file under Broker Configs. enable.auto.commit is a consumer config.

Not sure I understand what you mean by "commit logs". The consumer group offsets are stored in a topic named __consumer_offsets. All topics are stored in segment files on the broker in log.dirs

You can use tools such as JMX Exporter + Prometheus + Grafana, or Cruise Control for general Kafka monitoring. Both should give you an idea about ISR status