LinkedIn
Twitter
This article addresses the common enterprise requirement of controlling internet egress traffic through a central firewall appliance and how this can be achieved with Apigee X.
We’ll build up the required components one by one to describe the different components and how they are used to route the Apigee runtime egress traffic via a firewall appliance. If you prefer a ready made example setup please skip to the end of this article where you’ll find instructions on how to provision the entire setup as a Terraform module.
Basic requirement of controlling internet egress
Many enterprise customers require all internet egress traffic to be routed through a central egress gateway that can act as a firewall and is able to log any internet facing traffic. This requirement should of course also be applied to the API platform to prevent the API gateway from becoming an unmonitored internet egress point. With Apigee hybrid, the API traffic is part of the customer’s Kubernetes cluster and can be controlled like any other egress traffic from the underlying worker nodes.
With Apigee X the situation is a bit different in that the API platform runs on Google-managed infrastructure, where the routes and firewalls are not directly operated by the customer. In the remainder of this article we will describe how you can still control all internet egress traffic even if you are using Apigee X inside a Google-managed tenant project.
The exact deployment and configuration of the firewall is product specific and outside of the scope of this article. There are a number of firewall appliances, many of which can be installed through the GCP marketplace that can be used for the purpose of filtering egress internet traffic. In this article we will use a firewall mock that is performing a Network Address Translation (NAT) on any internet egress traffic before routing it to the default internet gateway. In your target architecture you will want to replace this VM with a proper firewall that can perform the actual decisions based on the IP packets.
Apigee X with Default Internet Egress Configuration
By default an Apigee X runtime has unrestricted internet egress access. This means that even if you decide not to expose your API ingress endpoints publicly they can still talk to any publicly facing API on the internet within a service callout or use them as an API target endpoint.
You can test the above by creating a new API proxy in your Apigee organization for https://mocktarget.apigee.net with the basepath /mock and deploying it to one of your environments.
When you call that proxy you should see a successful response like the following:
$ curl https://api.example.com/mock -I
HTTP/2 200
x-powered-by: Apigee
…Removing Internet Egress from Apigee
To prevent Apigee from directly calling an internet endpoint you can enable VPC Service Controls on the service networking connection.
$ gcloud services vpc-peerings enable-vpc-service-controls \
--network=$NETWORK --project $PROJECT_ID \
--service servicenetworking.googleapis.com
Operation "operations/…" finished successfully.
Under the hood this deletes the default internet egress route in the Apigee tenant project which means that the Apigee runtime won’t be able to connect to endpoints on the public internet.
Because the default internet route on the Apigee tenant project is not exported to the peered VPC network we do not see the absence of the default internet route directly but we can see the effect of this change when we try to call the API proxy from before:
$ curl https://api.example.com/mock -I
HTTP/2 503The response body contains an Apigee error message indicating that the request to the target endpoint had timed out:
$ curl https://api.example.com/mock
{"fault":{"faultstring":"The Service is temporarily unavailable","detail":{"errorcode":"messaging.adaptors.http.flow.ServiceUnavailable"}}}`If we didn’t want to connect to any APIs on the public internet we could stop here.
Providing Internet Egress through a Customer-Managed Firewall Appliance
Now that we have removed the internet egress route from the Apigee runtime we want to re-enable it by routing all internet egress traffic through a customer managed firewall.
As discussed initially, the configuration of the firewall appliance is specific to the firewall that you have in place. For this demonstration we will therefore mock the firewall with a simple virtual machine that performs a NAT operation through iptables.
To create our mock firewall we run the following (If using these commands, please ensure the environment variables referenced have first been appropriately defined):
$ gcloud compute instances create mock-fw --project=$PROJECT_ID \
--zone=$ZONE --machine-type=e2-micro --network-interface=subnet=$SUBNET
$ gcloud compute firewall-rules create mock-fw-ssh --allow tcp:22 \
--network $NETWORK --project $PROJECT_ID
$ gcloud compute firewall-rules create mock-fw-https --allow tcp:443 \
--network $NETWORK --project $PROJECT_IDWe then set up the IP forwarding and iptables NATing as follows
$ gcloud compute ssh mock-fw --zone $ZONE \
--command="sudo sysctl -w net.ipv4.ip_forward=1 && sudo iptables -t nat -A POSTROUTING -j MASQUERADE"
Now we have to tell the Apigee runtime to route all internet egress traffic through the firewall mock that we just created. We do this by creating one of the following:
Option A: If you have a default internet egress route in your VPC network and want to continue to allow direct internet egress for all resources in your VPC, then you create a route that can be exported to the Apigee tenant project and that has a lower priority order (higher priority value) than the existing default internet route:
$ gcloud compute routes create internet-via-fw --project=$PROJECT_ID \
--network=$NETWORK --priority=1001 --destination-range=0.0.0.0/0 \
--next-hop-instance=mock-fw --next-hop-instance-zone=$ZONEOption B: If you deleted the default internet route because you want all resources in your network to connect to the internet through the firewall appliance, you can create the following two routes where the first one is telling all the resources to route their traffic to the firewall appliance and the second rule only targets the firewall to send its traffic to the default internet gateway:
$ gcloud compute routes create internet-via-fw --project=$PROJECT_ID --network=$NETWORK \
--priority=600 --destination-range=0.0.0.0/0 --next-hop-instance=mock-fw \
--next-hop-instance-zone=$ZONE
$ gcloud compute routes create internet-gw-for-fw --project=$PROJECT_ID \
--network=$NETWORK --priority=599 --tags=mock-fw --destination-range=0.0.0.0/0 \
--next-hop-gateway=default-internet-gatewayOnce the routes are configured you need to ensure that the custom route is exported to the peered Apigee tenant project.
$ gcloud compute networks peerings update servicenetworking-googleapis-com \
--network=$NETWORK --project=$PROJECT_ID --export-custom-routes
With this configuration we can try calling our proxy again:
$ curl https://api.example.com/mock -I
HTTP/2 200
x-powered-by: Apigee
…
If everything worked correctly we are now routing the Apigee egress traffic to the firewall appliance where we can perform the required checks and logging operations before routing to the internet endpoint.
To validate that the request is processed by the mock firewall you can use the following command (replace the CIDR range with your Apigee peering range) to run tcpdump while you call your mock proxy.
gcloud compute ssh mock-fw --zone $ZONE --project=$PROJECT_ID \
--command "sudo tcpdump port 443 and net 10.0.0.0/22"In the tcpdump you should see the request coming from the Apigee runtime instance that the firewall then forwards to the internet gateway as well as the response that the firewall sends back to the Apigee runtime.
Accessing Google APIs (Optional)
In the previous section we confirmed that we can build a routing path via a firewall appliance to a public endpoint on the internet. But what happens when our endpoint is a Google API? Google provides a mechanism that allows for private Google access. To validate that Apigee is able to use private Google access to access Google APIs, we create a proxy for the Google Storage API to list buckets (https://storage.googleapis.com/storage/v1/) and set the following HTTPTargetConnection to leverage Google service account authentication:
And use Google Authentication to authenticate against the storage API.
<HTTPTargetConnection>
<Authentication>
<GoogleAccessToken>
<Scopes>
<Scope>https://www.googleapis.com/auth/cloud-platform.read-only</Scope>
</Scopes>
</GoogleAccessToken>
</Authentication>
<URL>https://storage.googleapis.com/storage/v1/b</URL>
</HTTPTargetConnection>To gain some insights into the behavior of the tenant project we also create an AssignMessage policy that we add to the TargetResponseFlow:
<AssignMessage continueOnError="false" enabled="true" name="debug-rtarget">
<Add>
<Headers>
<Header name="x-target-ip">{target.ip}</Header>
<Header name="x-target-cn">{target.cn}</Header>
<Header name="x-target-host">{target.host}</Header>
</Headers>
</Add>
</AssignMessage>
If we deploy this proxy with a service account that is authorized to list storage buckets, we can call the proxy and see that it uses an IP from the private Google access range for restricted.googleapis.com which is 199.36.153.4/30:
$ curl https://api.example.com/buckets?project=$PROJECT_ID -I
HTTP/2 200
…
x-target-ip: 199.36.153.4
x-target-cn: storage.googleapis.com
x-target-host: storage.googleapis.comWe can also ensure that this call is not routed via the firewall by running tcpdump on the firewall mock as described before or by deleting the internet route for the Apigee runtime and validating that the call still completes.
Example automation
This example is also provided as an end to end pattern in the Apigee Terraform modules repository.
To deploy this example including an Apigee Eval organization execute the following steps:
git clone https://github.com/apigee/terraform-modules.git
cd terraform-modules/samples/x-controlled-internet-egress
terraform init
terraform apply --var-file=./my-config.tfvars -var "project_id=$PROJECT_ID"
# wait for the org and instance to provision
gcloud services vpc-peerings enable-vpc-service-controls \
--network=NETWORK --project=PROJECT_IDYou can now deploy a mock proxy as described above and verify that you can reach it even though you have VPC SC enabled on your service networking connection.
