cannot hit pod in kubernetes cluster from other pod but can from ingress

Question

I'm able to hit a pod from outside my k8s cluster using an ingress but cannot from within the cluster and am getting a "connection refused" error. I tried to shell into the pod that's refusing connections and run the following curls which work just fine when running in my local/host environment:

• curl localhost:4000/api/v1/users
• curl 127.0.0.1:4000/api/v1/users
• curl 0.0.0.0:4000/api/v1/users
• curl :4000/api/v1/users

to no avail. The cluster ip is 10.99.224.173 but that times out and I'd prefer not to bypass dns since this is dynamically assigned by k8s. And it's not working anyway. The service is a nodejs based one. I can add more information but figured I'd try to err on the side of too little information than too much. To isolate the issue as being a k8s problem, I've run the two services locally outside of k8s with no issues. I think a good starting point would be to identify why I can't curl to the server from within the same pod. Thanks!

EDIT 2: closing the cluster from skaffold and re-running skaffold dev resolved this issue and I'm now able to run the following just fine:

• curl localhost:4000/api/v1/users
• curl 127.0.0.1:4000/api/v1/users
• curl 0.0.0.0:4000/api/v1/users
• curl :4000/api/v1/users

I found that the tchannel-node library does not accept 0.0.0.0 as a valid ip address to listen to, and the closest I can pass is 127.0.0.1. Unfortunately, this means that calling to the cluster ip 10.99.224.173:9090 will never be registered by the server as 127.0.0.1:9090 the way 0.0.0.0:9090 will. I'm wondering how I can fix my understanding to pass the correct ip address.

EDIT (requested yaml files):

• client

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tickets-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tickets
  template:
    metadata:
      labels:
        app: tickets
    spec:
      containers:
        - name: tickets
          image: mine/tickets-go
---
apiVersion: v1
kind: Service
metadata:
  name: tickets-svc
spec:
  selector:
    app: tickets
  ports:
    - name: tickets
      protocol: TCP
      port: 4004
      targetPort: 4004

• server that refuses connections

apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: auth
  template:
    metadata:
      labels:
        app: auth
    spec:
      containers:
        - name: auth
          image: mine/auth
          env:
            - name: PORT
              value: "4000"
            - name: TCHANNEL_PORT
              value: "9090"
---
apiVersion: v1
kind: Service
metadata:
  name: auth-svc
spec:
  selector:
    app: auth
  ports:
    - name: auth
      protocol: TCP
      port: 4000
      targetPort: 4000
    - name: auth-thrift
      protocol: TCP
      port: 9090
      targetPort: 9090

• ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-svc
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - host: foo.com
      http:
        paths:
          - path: /api/v1/users/?(.*)
            backend:
              service:
                name: auth-svc
                port:
                  number: 4000
            pathType: Prefix
          - path: /api/v1/tickets/?(.*)
            backend:
              service:
                name: tickets-svc
                port:
                  number: 4004
            pathType: Prefix

question from:https://stackoverflow.com/questions/65873736/cannot-hit-pod-in-kubernetes-cluster-from-other-pod-but-can-from-ingress

Answer 1

Waitting for answers