kubernetes - Egress IP address selection

Question

We are running a SaaS service that we are looking to migrate to Kubernetes, preferably at one of the hyperscalars. One specific issue I have not yet found a clean solution for is the need for Egress IP address selection from within the application.

We deal with a large amount of upstream providers that have access control and rate limiting based on source IP adres. Also a partition of our customers are using their own accounts with some of the upstream providers. To access the upstream providers in the context of their account we need to control the source IP used for the connection from within the application.

We are running currently our services in a DMZ behind a load balancer, so direct network interface selection is already impossible. We use some iptables rules on our load balancers/gateways to do address selection based on mapped port numbers. (e.g. egress connections to port 1081 are mapped to source address B and target port 80, port 1082 to source address C port 80)

This however is quite a fragile setup that also does not map nicely when trying to migrate to more standardized *aaS offerings.

Looking for suggestions for a better setup.

Answer 1

One of the things that could help you solve it is Istio Egress Gateway so I suggest you look into it.

Otherwise, it is still dependent on particular platform and way to deploy your cluster. For example on AWS you can make sure your egress traffic always leaves from predefined, known set of IPs by using instances with Elastic IPs assigned to forward your traffic (be it regular EC2s or AWS NAT Gateways). Even with Egress above, you need some way to define a fixed IP for this, so AWS ElasticIP (or equivalent) is a must.