Seamless Container Traffic Management: Azure Application Gateway's new ingress capability

1. What is Application Gateway for Containers?

The newest iteration in the lineage of Application Gateways aims to provide a combination of features already included in the well-known regular Application Gateway and the Application Gateway Ingress Controller, while providing some significant improvements and upgrades over the aforementioned offerings. 

As a Gateway, the new preview feature (like its predecessors) provides OSI layer 7 traffic management and load balancing capabilities; however, it now fully catered towards workloads running in a Kubernetes Cluster. Furthermore, being a Microsoft-managed solution, the Application Gateway for Containers takes away the platform-level efforts required to keep the Ingress Controller up and running. It eliminates the need for periodic patching and maintenance of the ingress controller.

The Application Gateway for Containers offers more scalability, allowing admins to create bigger backend pools and listeners. There is nearly instant alignment between the configuration made within the cluster and the Application Gateway resource located in Azure. Furthermore, it allows the Gateway API to configure a richer set of features, including header-based routing, weighted traffic splitting, and much more.

2. AGfC vs AGIC

As mentioned, the new Application Gateway for Containers (AGfC) allows admins to leverage the Gateway API for Kubernetes, instead of being limited to the Ingress API, which is exposed to users of the older Application Gateway Ingress Controller (AGIC). Both provide a couple of useful routing capabilities. However,  the Gateway API is more extensive than the Ingress API. Below is a comparison of both SKUs:

3. How does it work?

There are a couple of key components that enable the external Application Gateway for Containers resource to route traffic towards the correct backend service. As seen from the design overview below, resources both inside and outside of the Azure Kubernetes Service (AKS) cluster need to be foreseen for this setup to work.

3.1 Outside AKS

Once an admin deploys the “Application Gateway for Containers” resource using the Azure Resource Manager (blue flow), a public endpoint is automatically created along with it. This endpoint will be the only interface used by the Application Gateway for incoming traffic. If needed, you can create more of these endpoints and link them to other gateways within the Kubernetes cluster. Although Azure provides you with a randomized domain name for this endpoint, you can also use your own DNS names.

The only real configuration the AGfC resource needs is an association to a Virtual Network, which is required for network line-of-sight towards the in-cluster gateway. You can either associate the resource directly to the AKS VNET, or use a peered VNET which has connectivity towards the AKS backend network. Because network connectivity between the ingress controller and the AKS backend is required in order to forward requests, the AKS Cluster should use Azure CNI as its networking type. Azure CNI deploys a virtual network in the environment, allowing for a peering with the Application Gateway for Containers instance. When the kubenet network type is chosen for the cluster, this is not the case, preventing a direct connection between AGfC and the Kubernetes backend resources.

Important to note however, Azure CNI Overlay network type is not supported either. Even though this network type also deploys a virtual network in the environment, the pods use natting on the node IP, which will not work with the AGfC service.

3.2 Inside AKS

The main configuration for routing happens within the AKS cluster. Because the Application Gateway for Containers feature uses the Kubernetes Gateway API for its routing configuration, the main setup is done through the Kubernetes API.

3.2.1 Internal Gateway

First of all, there is the Internal Gateway. This gateway will act as a HTTP Listener which is linked with the outside Application Gateway. Referring to the architectural drawing above, you can see that the link between the outside Application Gateway and the Internal Gateway form the bridge between the AKS Cluster and the AGfC resource located in Azure.

The manifest for this gateway:

As illustrated, there is little to no configuration done on the gateway itself. Only the specific information related to the Azure Application Gateway resource and the internal HTTP listener is stored here to create the link between both resources.

The deployed resource in AKS:

3.2.2 HTTPRoute

For the above gateway to know where to route the different requests to, it needs a HTTPRoute. HTTPRoutes can be used to configure the different routing capabilities such as traffic splitting and load balancing.

An example of how a HTTPRoute can be configured within the AGfC context:

apiVersion: gateway.networking.k8s.io/v1beta1 

kind: HTTPRoute 

metadata: 

  name: httproute-lbdemo 

  namespace: workloads 

spec: 

  parentRefs: 

  - name: gateway-01 

  hostnames: 

  - "xxxxxxxxxxxxxxxxxx.fz32.alb.azure.com" 

  rules: 

  - backendRefs: 

    - name: backendservice-lbdemo-1 

      port: 80 

      weight: 50 

    - name: backendservice-lbdemo-2 

      port: 80 

      weight: 50 

The snippet above shows the configuration for the traffic splitting capability. Traffic is load balanced over the backends listed, according to their weight. The manifest for other capabilities will look slightly different, however the main structure remains the same.

As seen in the “parentRefs” spec, the HTTPRoute is a child resource of the gateway. The hostnames property contains a list of domain names, which correlate to the public endpoints where the Application Gateway for Containers will receive all requests at. It is important this is configured correctly, as the HTTPRoute will only apply to the traffic coming from this specific hostname(s).

In the illustration above, the importance of correct hostnames is clarified. In this case, whenever a user browses the “demo1.com” frontend, they will be routed towards “backend1”, because HTTPRoute1 is scoped on that specific hostname.

If a user browses the “demo2.net” frontend, the traffic will NOT be routed to any backend, and the user will be shown a blank page. This is because the gateway doesn’t know how to route the traffic coming from demo2.net.

Finally, no traffic will reach backend3, as there is no frontend foreseen for this backend and no HTTPRoute has been scoped on this URL (assuming there is no other ingress controller installed in the cluster).

The deployed resource in AKS:

3.2.3 Securing the Service

In this demo configuration, the ingress controller will be listening on port 80. This isn’t a secure way of setting up the environment and might be considered an open invitation to malicious actors. For production workloads, using HTTP is never advised and should always be converted to HTTPs (HTTP over TLS/SSL).

To secure this sample, the listener configured on the gateway resource (section 3.2.1) needs to be replaced by a HTTPs type listener (port 443). By doing so, traffic between the ingress controller and the user’s client device, will be encrypted.

To further improve the security posture of the environment, we can use a feature provided with the AGfC resource.:

The Application Gateway for Containers supports Backend mTLS, meaning that traffic between the ingress controller and the backend pods is also encrypted. By configuring mTLS, both ingress controller and backend pods will authenticate each other (hence mTLS being the abbreviation for mutual TLS). This prevents man-in-the-middle attacks and provides full end-to-end encryption between the client device and the backend handling the request. However, this is more complex as it requires configurations on the client device as well and should only be considered for managed client devices or services.

To configure this encryption, a BackendTLSPolicy needs to be created in the namespace, specifying the backend targets, along with the certificate information.

3.2.4 The Ingress Controller in Action

For demo purposes, 2 backend deployments were created to show how the traffic splitting works. These are very basic deployments, showing some details about the pod that is being targeted.

Deployments in AKS:

Pods in AKS:

Both delployments have their opwn backend service, exposing the pods to the internal network:

The ingress controller is configured through its HTTPRoute to split the traffic over both deployments, using a 50/50 weight distribution (as seen from the manifest above).

We know what frontend URL our Application Gateway for Containers is listening on. Browsing to this URL should show us the information about the pod we land on. Because we have setup the HTTPListener to scope on this specific URL, we should not be getting a blank page:

As can be seen from this first request, we landed on the “backend-lbdemo-1- 8d778bc44-bc98t” pod, which is indeed the pod provided through the first deployment.

Upon refreshing the webpage a couple of times, the following result is shown:

The request reached the 2nd pod, originating from the “backend-lbdemo-2” deployment.

In theory, each request has a 50/50 chance of landing on either one of the deployments. Because this is a demo, there is not really much use in splitting the traffic like this, however, when we want to test a new production environment and expose it to only a subset of the users, a weight-distribution of 95/5 can be set for example. By doing so, we can test the new deployment without impacting all active users. This is called a canary roll-out.

Because the Application Gateway for Containers is an Azure resource, we get metrics which give some insight into the usage and health of the environment. The various requests from this demo are showing up like this:

4. Why is this important for us

As an Azure Managed Service provider, we as The Collective strive towards simplifying the maintenance and setup of all aspects of our customers’ environments. In using this new Application Gateway for Containers feature, we provide our customers with a more extensive service by leveraging the benefits of these PaaS offerings, in contrary to a self-managed solution.

4.1 Quicker deployments

Because this new feature allows for deployment using ARM, Bicep, Terraform and even PowerShell, we can use existing templates to speed up the deployment process, and shorten the overall project lead time. The self-managed variants of Ingress Controllers also allow for code deployments, however it is a lot harder to provide the customer with a deployment catered to their environment, when the resource depends on the Kubernetes architecture, which will be different for all environments.

Furthermore, the configuration done within the Kubernetes cluster (ALB, HTTPRoutes, etc.) is very modular and allows us to provide a quick and on-demand service for our customers. Using the manifest files we have created for each of the AGfC components, we can quickly act whenever a customer requests a modification in their environment.

This all allows the customer to focus on the workloads running in their cluster, rather than spending time working out how to publish them securely.

4.2 Security baselines

There is a significant increase in visibility when it comes to securing the environment. Because the new Application Gateway is visible as resource in the Azure Portal, the team can get a quick overview of how the service is configured. Having the AGfC visible in the Azure Portal also allows us to scope our security baseline policies to this resource, further ensuring no unwanted or unsecure changes are made to the gateway. We can set RBAC permissions on the gateway for a more modular approach when it comes to access management..

In contrary to a traditional in-cluster ingress controller setup, we are able to treat the resource like any other within the Azure platform, without having to delve into the Kubernetes cluster for this information.

Overall, the new Application Gateway for Containers lets us unify the way these resources are secured and managed, allowing for more alignment of these security principles with the rest of the Azure environment.

4.3 Monitoring

Metric and log data is also made available directly through Azure Monitor, allowing us to setup alerting rules and notify the team whenever an issue arises. This is important because it eliminates the need for Kubernetes specific monitoring/alerting such as Prometheus for a resource as vital as the Application Gateway. Furthermore, there is no need for additional development when it comes to routing these alerts to our operations personnel, as this is already setup for the other Azure resources.