How to configure hub/spokes egress through a single VPC?

[marioapra]

Hello all, I'm trying to figure out how I can have all the outbound traffic to go through a single vpc.

Requirements:
- all egress traffic goes through a single cloud NAT in a vpc (called hub)
- traffic between two different spokes needs to go throught hub.
- I don't want to change my applications running in the spokes (for example letting them be aware that they are behind a proxy)

I want to have a hub-spoke architecture.  The spokes will be located in a different project (and different vpc).
For all the spokes, the default route that goes to the default internet gateway will be deleted, together with the cloud nat created by default.

The hub and spokes are currently connected to each other via NCC (Network connectivity center), but I'm open to connect them differently (vpc peering, vpn, etc).

My biggest blocker is that when I try to create a route in the spoke VPC and seeting any resource in the hub vpc as a next hop, it doesn't let me.

What I've tried so far:
- create a Secure Web Proxy in the hub and configure the spoke to use the proxy.  The downside of this is that every application in the spoke needs to be aware that it's running behind a proxy, so that's not viable.  For example this is required to work: `curl example.com --proxy private-web-proxy-ip:443`
- creating a VM in the hub (with port foward enabled) that should be used for NAT.  That machine runs "sysctl -w net.ipv4.ip_forward=1 && iptables -t nat -A POSTROUTING -o $(/sbin/ifconfig | head -1 | awk -F: {\'print $1\'}) -j MASQUERADE" and it's running behind a internal load balancer, so it can be auto-scaled easily.  (inspired by https://cloud.google.com/bare-metal/docs/bms-setup#bms-access-internet-vm-nat).  It didn't work for me because I'm still using NCC, so the routes created in the hub are not propagated to the spokes.  When I try to set the vm ip or the load balancer as a next hop, it always complains saying that the range is outside of the network
- My next attempt will be to create a vm in the spoke to talk with the vm in the hub.  resource in spoke that wants internet -> vm in spoke -> vm in hub -> cloud nat -> internet.  This is quite annoying and it will be costly to have one vm per spoke