apache flink - Taskmanager can't connect to jobmanager leader in kubernetes with High availability

Question

I can't seem to get my Taskmanager to connect to Jobmanager in Kubernetes HA mode. Has anyone seen anything similar? The following settings are being used (Same on Jobmanager and Taskmanager)

jobmanager.rpc.address: my-jobmanager
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
kubernetes.cluster-id: flinko
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.storageDir: s3://<hidden>/flink/recovery
kubernetes.namespace: my-namespace
kubernetes.service-account: my-serviceaccount

Here are the log entries which i think are of interest.

2021-01-26 14:43:29,177 INFO  org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='flinko-resourcemanager-leader'}.
2021-01-26 14:43:29,177 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils           [] - Trying to select the network interface and address to use by connecting to the leading JobManager.
2021-01-26 14:43:29,177 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils           [] - TaskManager will try to connect for PT10S before falling back to heuristics
2021-01-26 14:43:39,179 WARN  org.apache.flink.runtime.net.ConnectionUtils                 [] - Could not find any IPv4 address that is not loopback or link-local. Using localhost address.
2021-01-26 14:43:39,179 INFO  org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Stopping DefaultLeaderRetrievalService.
2021-01-26 14:43:39,179 INFO  org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver [] - Stopping KubernetesLeaderRetrievalDriver{configMapName='flinko-resourcemanager-leader'}.

The ConfigMap for resourcemanager-leader;

- apiVersion: v1
  data:
    address: akka.tcp://flink@my-jobmanager:6123/user/rpc/resourcemanager_0
    sessionId: 7357bb1c-4beb-4455-8344-e72347245d1c
  kind: ConfigMap
  metadata:
    annotations:
      control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"b2a0daac-5fe6-4704-9690-9591c9dde437","leaseDuration":15.000000000,"acquireTime":"2021-01-26T14:42:52.139000Z","renewTime":"2021-01-26T14:54:22.620000Z","leaderTransitions":139}'
    creationTimestamp: "2021-01-26T13:26:20Z"
    labels:
      app: flinko
      configmap-type: high-availability
      type: flink-native-kubernetes
    name: flinko-resourcemanager-leader
    namespace: my-namespace
    resourceVersion: "512789629"
    selfLink: /api/v1/namespaces/my-namespace/configmaps/flinko-resourcemanager-leader
    uid: 15171ee3-f604-49b7-88cf-2aaa752118df

question from:https://stackoverflow.com/questions/65903691/taskmanager-cant-connect-to-jobmanager-leader-in-kubernetes-with-high-availabil

Answer 1

Waitting for answers