IPs are masqueraded by default in Autopilot to use the node IP for egress traffic. which is just a fancy way of saying, the Pod traffic looks like it comes from the node’s IP.
This is handy when using non-RFC1918 ranges in GKE to avoid IP exhaustion, since your Node IP range is typically in RFC 1918 space which is what other services may be expecting. However, GKE does not masquerade traffic to internal addresses (like 10.0.0.0/8) by default. If you have a service that is expecting IPs in the RFC 1918 range, you may need to adjust the masquerade rules to include those destinations.
In this walkthrough, I demonstrate how to modify the egress masquerading rules.
To test, I’m going to create a cluster with Nodes in the default network (10.0.0.0/8), and IPs in non-RFC 1918 space:
gcloud container clusters create-auto autopilot-21 \
--cluster-ipv4-cidr "240.62.0.0/17" \
--region us-west1
And create a Cloud SQL database on an internal IP
$ ROOT_PASSWORD="your database password"
$ gcloud sql instances create test-instance \
--database-version=MYSQL_5_6 \
--tier=db-g1-small \
--region=us-west1 \
--no-assign-ip \
--network default \
--root-password=YOUR_ROOT_PASSWORD
By default, this database will accept connections only from RFC1918 addresses. Since our IPs are in the 240.0.0.0/4 range, if we try to connect to it, it will time out:
$ MYSQL_HOST=10.81.208.5
$ kubectl run my -it --rm --restart=Never \
--pod-running-timeout=3m \
--image mysql -- mysql -h $MYSQL_HOST -P 3306 -u root -p
ERROR 2003 (HY000): Can't connect to MySQL server on '10.81.208.5:3306' (110)
Now we could simply add 240.0.0.0/4 to the authorized networks for this SQL instance, but for this demo, let’s turn on masquerading instead.
Following the instructions, edit the egress NAT policy. This policy contains a list of ranges where IP Masquerading is not applied. We want to remove the IP of our MySQL server from this list.
$ kubectl get -o yaml egressnatpolicies default
apiVersion: networking.gke.io/v1
kind: EgressNATPolicy
metadata:
creationTimestamp: "2023-10-04T22:59:12Z"
generation: 1
name: default
resourceVersion: "3374"
uid: f6c3843c-0a28-44fd-bc2c-e7fc42019665
spec:
action: NoSNAT
destinations:
- cidr: 10.0.0.0/8
- cidr: 172.16.0.0/12
- cidr: 192.168.0.0/16
- cidr: 240.0.0.0/4
- cidr: 192.0.2.0/24
- cidr: 198.51.100.0/24
- cidr: 203.0.113.0/24
- cidr: 100.64.0.0/10
- cidr: 198.18.0.0/15
- cidr: 192.0.0.0/24
- cidr: 192.88.99.0/24
status: {}
The line to remove is 10.0.0.0/8. To edit in-place:
kubectl edit egressnatpolicies default
Note: Instead of removing the entire 10.0.0.0/8 range, you could be selective by adding a bunch of subranges. Using a CIDR range calculator for example, I came up with this list to replace 10.0.0.0/8.
- cidr: 10.0.0.0/10
- cidr: 10.64.0.0/12
- cidr: 10.80.0.0/16
- cidr: 10.82.0.0/15
- cidr: 10.84.0.0/14
- cidr: 10.88.0.0/13
- cidr: 10.96.0.0/11
- cidr: 10.128.0.0/9It’s enough for this test though to just to delete the 10.0.0.0/8 line and update.
NOTE: You’ll need to wait 2-3 minutes for this to take effect.
After waiting for the IP Masq rules to sync, when we re-run the container, the connection is accepted, and we can see the node’s IP in the output (note the access denied error and the IP of the client, instead of the previous timeout):
$ kubectl run my -it --rm --restart=Never \
--pod-running-timeout=3m \
--image mysql -- mysql -h $MYSQL_HOST -P 3306 -u root -p
If you don't see a command prompt, try pressing enter.
ERROR 1045 (28000): Access denied for user 'root'@'10.138.0.90' (using password: YES)
Summary
IP masquerading is a convenient technique for making it appear that Pod traffic is coming from the Node IP. This can be especially useful when the traffic needs to originate from a RFC 1918 IP, like those used for Nodes.