You may have already heard of the novel Kubernetes Gateway API. The Gateway API – on which work started in 2019 – is thought of as an alternative for Ingress resources. It introduces several new resource types like Gateways and Routes which in sum make up what used to be just Ingress. It is much more flexible and extensible and will make deploying applications that don't have HTTP ingress easier.
In this blog post I will be sharing a hands-on-replacement of classic Ingress with the new Gateways. I first show you a setup with Ingress and then will make the same services accessible with a Gateway.
But first, let's look at a few important design principles that the Gateway API builds on and which are meant to improve Ingress:
You see, that a lot about this new API is focusing on removing restrictions that Ingress have but preserving that good parts. A really important part is splitting up the Ingress resource into several new resources:
The current Gateway API is a beta version and there can be no recommendation to generally use it yet. Especially, since the support of implementations of the API like NGINX is not yet complete. Though there is no plan to replace the Ingress API anytime, there are advantages of Gateways over Ingresses. Next, I am going show you how to replace a basic NGINX Ingress with it.
For our example we are going to setup an application that listens at one hostname and consists of two services. We want to split the traffic coming to that hostname by a path prefix to our two services. Each service is fulfilled by a pod.
So first look at the classic Ingress architecture:
We will replace the Ingress with a Gateway and attach two HTTPRoutes to it, one for each path prefix.
We are going to use Kubernetes in Docker (KIND) to create a local cluster for our testing purposes. If haven't installed it yet, please do. You will also need kubectl to follow along.
All resources and yaml configs are available at this GitHub repository: https://github.com/benchkram/k8s-gateway-example. Download or clone it to try out the example setups.
Create a cluster and configure to use it:
kind create cluster --name ingress-example kubectl config use-context kind-ingress-example
We are using an NGINX Ingress controller for our example, let's deploy it:
kubectl apply -f nginx-ingress/nginx-ingress-controller.yaml
We are deploying two example pods, each with a service. The yaml shows one of the two pairs exemplary:
kind: Pod apiVersion: v1 metadata: name: pod-a labels: app.kubernetes.io/name: service-a spec: containers: - command: - /agnhost - netexec - --http-port - "8080" - --http-override - "/hostname" image: registry.k8s.io/e2e-test-images/agnhost:2.43 name: pod-a --- kind: Service apiVersion: v1 metadata: name: service-a spec: selector: app.kubernetes.io/name: service-a ports: - port: 8080
kubectl apply -f pods-service-manifest.yaml
The classic approach to date is to make the services accessible with an ingress. The example ingress that has two paths with prefix matching which each point to one of our services. You also see that the capabilities of the Ingress resource alone is not enough to specify the URL rewriting. We need the annotations field to pass custom configuration to the controller.
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 spec: rules: - http: paths: - pathType: Prefix path: /service-a(/|$)(.*) backend: service: name: service-a port: number: 8080 - pathType: Prefix path: /service-b(/|$)(.*) backend: service: name: service-b port: number: 8080
kubectl apply -f ingress-manifest.yaml
Use port-fording to bind the Ingress controller HTTP port to your localhost and test if the pods are reachable:
# find out what name the controller pod has kubectl get pods -n ingress-nginx kubectl port-forward -n ingress-nginx ingress-nginx-controller-<name> 8080:80 curl localhost:8080/service-a # outputs: pod-a curl localhost:8080/service-b # outputs: pod-b
Let's create a new cluster for the analogue example with the Gateway
kind create cluster --name gateway-example kubectl config use-context kind-gateway-example
We're going to stay with NGINX, but use the NGINX Gateway implementation next. Let's first install the general Kubernetes Gateway API:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.6.2/standard-install.yaml
Next we install the NGINX Gateway Controller and the GatewayClass.
apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: nginx spec: controllerName: k8s-gateway.nginx.org/nginx-gateway-controller
kubectl create namespace nginx-gateway kubectl create configmap njs-modules --from-file=nginx-gateway/httpmatches.js -n nginx-gateway kubectl apply -f nginx-gateway/nginx-gateway-controller.yaml kubectl apply -f gatewayclass-manifest.yaml # check it's working kubectl get pods -n nginx-gateway kubectl get gatewayclasses.gateway.networking.k8s.io
And again our pods and services A and B:
kubectl apply -f pods-service-manifest.yaml
Instead of deploying the Ingress we will now add the Gateway instead. Thinking in roles intended for this, deploying the Gateway would be the job of a cluster operator. This is a minimal working example that uses the NGINX GatewayClass that we already installed. We define one HTTP Listener with no hostname, so we catch all incoming traffic. You can have multiple Listeners each with a different host:port combination.
apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: combined-gateway spec: gatewayClassName: nginx listeners: - name: http protocol: HTTP port: 80
We are using HTTPRoutes and attach them to the Gateway with the
name references the name of the Gateway and
sectionName the name of the Listener in that Gateway.
apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: service-a spec: parentRefs: - name: combined-gateway sectionName: http hostnames: - localhost rules: - matches: - path: type: PathPrefix value: /service-a # filters: # - type: URLRewrite # urlRewrite: # path: # type: ReplacePrefixMatch backendRefs: - name: service-a port: 8080
# apply gateway and routes kubectl apply -f gateway-manifest.yaml
The commented section is how Gateways specify the URL rewrite that we know from the Ingress annotation above. It is commented because the URLRewrite filter is currently not supported by the NGINX Gateway. Have I already told you that this NGINX Gateway controller is also a beta version 😀😅? To make the example work anyway I configured the pods to ignore the prefix with which they are called.
# find out what name the controller pod has kubectl get pod -n nginx-gateway kubectl port-forward -n nginx-gateway <gateway-name> 8080:80 curl localhost:8080/service-a # outputs: pod-a curl localhost:8080/service-b # outputs: pod-b
You have seen how easy it is to replace Ingress with Gateway while keeping your services and pods. The next step from here is exploring the new load balancing features of Gateways which allow e.g. canary releases and using other types of Routes than just HTTPRoutes. Also make sure to check out what implementations of the Gateway API already exist.
Lastly, I want to share some of the debugging tools that I used. As always when you learn new technologies it is important to understand the trial-and-error-loop. While you usually find a lot of information about problems in your Kubernetes cluster in the events, that didn't work for me with Gateways. I started to read out the status fields of the Gateways and HTTPRoutes. They will show you misconfigurations and display e.g. how many routes are attached to a Gateway listener. You find the status when you execute:
kubectl describe gateway kubectl describe route
Do you already know k9s? It is a terminal UI for Kubernetes and I use it a lot if there is no Kubernetes Dashboard or other means of management tool. Also very handy for a quick look into a KIND cluster.
We are a Software Agency from Stuttgart, Germany and can support you on your journey to deliver outstanding user experiences and resilient infrastructure. Reach out to us.
Make sure to check out our build tool for large multi-language applications: https://bob.build