I trying to install elastic as cluster on three environments, where previuosly was installed and working.
i have a elasticsearch 7.8.1 zip package and this i have put and extracted on three machines (im in Windows ) and i set the same elasticsearch.yml file for all three nodes as:
cluster.name: elk-elastic-cluster
node.name: elk-elastic-cluster-node-1
path.data: G:elasticsearchdata
path.logs: G:elasticsearchlogs
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["pod-elk-006", "pod-elk-007", "pod-elk-008"]
cluster.initial_master_nodes: ["elk-elastic-cluster-node-1", "elk-elastic-cluster-node-2", "elk-elastic-cluster-node-3"]
logger.org.elasticsearch.discovery: TRACE
logger.org.elasticsearch.cluster.coordination: TRACE
when i start elastic on every machines i have in the log of first node:
[2021-01-21T17:01:11,209][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elk-elastic-cluster-node-1] master not discovered or elected yet, an election requires at least 2 nodes with ids from [UlnGHt3XRsSZxZH0YEwVJw, m0QefzOjS7Kq4H0gbly-3A, iPRCXAlVTxCgQoNedae-7w], have discovered [{elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [[::1]:9300, 10.168.333.102:9300, 10.168.333.103:9300] from hosts providers and [{elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5, last-accepted version 947 in term 5
[2021-01-21T17:01:11,859][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] probing master nodes from cluster state: nodes:
{elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, local
[2021-01-21T17:01:11,859][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] startProbe(10.168.333.101:9300) not probing local node
[2021-01-21T17:01:11,859][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-006] to [10.168.333.101:9300, [::1]:9300]
[2021-01-21T17:01:11,859][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-007] to [10.168.333.102:9300]
[2021-01-21T17:01:11,859][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-008] to [10.168.333.103:9300]
[2021-01-21T17:01:11,859][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] probing resolved transport addresses [[::1]:9300, 10.168.333.102:9300, 10.168.333.103:9300]
[2021-01-21T17:01:11,859][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] Peer{transportAddress=[::1]:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2021-01-21T17:01:11,859][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] opening probe connection
[2021-01-21T17:01:11,869][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] opened probe connection
[2021-01-21T17:01:11,870][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] handshake successful: {elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}
[2021-01-21T17:01:11,871][DEBUG][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] Peer{transportAddress=[::1]:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [elk-elastic-cluster-node-1][10.168.333.101:9300] local node found
at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1$1$1.innerOnResponse(HandshakingTransportAddressConnector.java:105) [elasticsearch-7.8.1.jar:7.8.1]
[...cutted here...]
[2021-01-21T17:01:12,865][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] probing master nodes from cluster state: nodes:
{elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, local
[2021-01-21T17:01:12,865][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] startProbe(10.168.333.101:9300) not probing local node
[2021-01-21T17:01:12,865][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-006] to [10.168.333.101:9300, [::1]:9300]
[2021-01-21T17:01:12,865][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-007] to [10.168.333.102:9300]
[2021-01-21T17:01:12,865][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-008] to [10.168.333.103:9300]
[2021-01-21T17:01:12,871][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] probing resolved transport addresses [[::1]:9300, 10.168.333.102:9300, 10.168.333.103:9300]
[2021-01-21T17:01:12,871][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] Peer{transportAddress=[::1]:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2021-01-21T17:01:12,871][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] opening probe connection
[2021-01-21T17:01:12,875][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] opened probe connection
[2021-01-21T17:01:12,876][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] handshake successful: {elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}
[2021-01-21T17:01:12,881][DEBUG][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] Peer{transportAddress=[::1]:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [elk-elastic-cluster-node-1][10.168.333.101:9300] local node found
at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1$1$1.innerOnResponse(HandshakingTransportAddressConnector.java:105) [elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1$1$1.innerOnResponse(HandshakingTransportAddressConnector.java:95) [elasticsearch-7.8.1.jar:7.8.1]
[...cutted here...]]
[2021-01-21T17:01:13,867][DEBUG][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] Peer{transportAddress=10.168.333.102:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [][10.168.333.102:9300] connect_timeout[3s]
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:972) ~[elasticsearch-7.8.1.jar:7.8.1]
[...cutted here...]]
log on second node:
[2021-01-21T17:08:38,525][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elk-elastic-cluster-node-2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elk-elastic-cluster-node-1, elk-elastic-cluster-node-2, elk-elastic-cluster-node-3] to bootstrap a cluster: have discovered [{elk-elastic-cluster-node-2}{nQ6ief-mSXa-wzly0xmCJw}{mqkTF7f2TRK_ybXHSHJJiw}{10.168.333.102}{10.168.333.102:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.168.333.101:9300, [::1]:9300, 10.168.333.103:9300] from hosts providers and [{elk-elastic-cluster-node-2}{nQ6ief-mSXa-wzly0xmCJw}{mqkTF7f2TRK_ybXHSHJJiw}{10.168.333.102}{10.168.333.102:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
on the third:
[2021-01-21T17:01:11,209][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elk-elastic-cluster-node-1] master not discovered or elected yet, an election requires at least 2 nodes with ids from [UlnGHt3XRsSZxZH0YEwVJw, m0QefzOjS7Kq4H0gbly-3A, iPRCXAlVTxCgQoNedae-7w], have discovered [{elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [[::1]:9300, 10.168.333.102:9300, 10.168.333.103:9300] from hosts providers and [{elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5, last-accepted version 947 in term 5
[2021-01-21T17:01:11,859][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] probing master nodes from cluster state: nodes:
{elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, local
[2021-01-21T17:01:11,859][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] startProbe(10.168.333.101:9300) not probing local node
[2021-01-21T17:01:11,859][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-006] to [10.168.333.101:9300, [::1]:9300]
[2021-01-21T17:01:11,859][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-007] to [10.168.333.102:9300]
[2021-01-21T17:01:11,859][TRACE][o.e.d.SeedHostsResolver ] [elk-elastic-cluster-node-1] resolved host [pod-elk-008] to [10.168.333.103:9300]
[2021-01-21T17:01:11,859][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] probing resolved transport addresses [[::1]:9300, 10.168.333.102:9300, 10.168.333.103:9300]
[2021-01-21T17:01:11,859][TRACE][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] Peer{transportAddress=[::1]:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2021-01-21T17:01:11,859][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] opening probe connection
[2021-01-21T17:01:11,869][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] opened probe connection
[2021-01-21T17:01:11,870][TRACE][o.e.d.HandshakingTransportAddressConnector] [elk-elastic-cluster-node-1] [connectToRemoteMasterNode[[::1]:9300]] handshake successful: {elk-elastic-cluster-node-1}{m0QefzOjS7Kq4H0gbly-3A}{VCyD3DGaTt2X2KALeTX95Q}{10.168.333.101}{10.168.333.101:9300}{dilmrt}{ml.machine_memory=8589328384, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}
[2021-01-21T17:01:11,871][DEBUG][o.e.d.PeerFinder ] [elk-elastic-cluster-node-1] Peer{transportAddress=[::1]:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.C