How To Replace Your K8s Ingress With the New Gateway API

Published at 5/5/2023

You may have already heard of the novel Kubernetes Gateway API. The Gateway API – on which work started in 2019 – is thought of as an alternative for Ingress resources. It introduces several new resource types like Gateways and Routes which in sum make up what used to be just Ingress. It is much more flexible and extensible and will make deploying applications that don't have HTTP ingress easier.

In this blog post I will be sharing a hands-on-replacement of classic Ingress with the new Gateways. I first show you a setup with Ingress and then will make the same services accessible with a Gateway.

What is the Gateway API?

But first, let's look at a few important design principles that the Gateway API builds on and which are meant to improve Ingress:

Role-orientation: Gateway API is role-based. The organizational roles they had in mind when designing this are: Infrastructure provider, Cluster operator and application developers. Previously for Ingresses, all of them had to work on the same Ingress resource. For Gateways, their responsibilities are split up into separate resources.
Portability: No improvement, but rather a continuation from Ingress. Gateway API also defines flexible conformance levels (mandatory, extended and custom).
Expressiveness: The Gateway API supports core functionality like header-based matching, traffic weighting, and more that were only possible through annotations in Ingress.
Extensibility: custom resources can extend the API at the appropriate level (e.g. more Route types).

You see, that a lot about this new API is focusing on removing restrictions that Ingress have but preserving that good parts. A really important part is splitting up the Ingress resource into several new resources:

GatewayClass: In Ingress there are Ingress classes which are implemented by Ingress controllers. Similarly Gateways have GatewayClass which is implemented by a Gateway controller.
Gateway: Gateways define one or more listeners which each have a hostname and port and define main entry points to the cluster.
Routes: There's in fact several Route resources which represent the different kind of ingresses that we typically see: HTTPRoute, TLSRoute, TCPRoute, UDPRoute, GRPCRoute. A Route which is attached to a parent Gateway and listener specifies the service to which traffic is forwarded.

The current Gateway API is a beta version and there can be no recommendation to generally use it yet. Especially, since the support of implementations of the API like NGINX is not yet complete. Though there is no plan to replace the Ingress API anytime, there are advantages of Gateways over Ingresses. Next, I am going show you how to replace a basic NGINX Ingress with it.

Example Architecture

For our example we are going to setup an application that listens at one hostname and consists of two services. We want to split the traffic coming to that hostname by a path prefix to our two services. Each service is fulfilled by a pod.

So first look at the classic Ingress architecture:

We will replace the Ingress with a Gateway and attach two HTTPRoutes to it, one for each path prefix.

Setting Up the Ingress Version

We are going to use Kubernetes in Docker (KIND) to create a local cluster for our testing purposes. If haven't installed it yet, please do. You will also need kubectl to follow along.

All resources and yaml configs are available at this GitHub repository: https://github.com/benchkram/k8s-gateway-example. Download or clone it to try out the example setups.

Create a cluster and configure to use it:

kind create cluster --name ingress-example
kubectl config use-context kind-ingress-example

NGINX Ingress Controller

We are using an NGINX Ingress controller for our example, let's deploy it:

kubectl apply -f nginx-ingress/nginx-ingress-controller.yaml

Pods and Services

We are deploying two example pods, each with a service. The yaml shows one of the two pairs exemplary:

kind: Pod
apiVersion: v1
metadata:
  name: pod-a
  labels:
    app.kubernetes.io/name: service-a
spec:
  containers:
    - command:
        - /agnhost
        - netexec
        - --http-port
        - "8080"
        - --http-override
        - "/hostname"
      image: registry.k8s.io/e2e-test-images/agnhost:2.43
      name: pod-a
---
kind: Service
apiVersion: v1
metadata:
  name: service-a
spec:
  selector:
    app.kubernetes.io/name: service-a
  ports:
    - port: 8080

kubectl apply -f pods-service-manifest.yaml

Classic Ingress

The classic approach to date is to make the services accessible with an ingress. The example ingress that has two paths with prefix matching which each point to one of our services. You also see that the capabilities of the Ingress resource alone is not enough to specify the URL rewriting. We need the annotations field to pass custom configuration to the controller.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - http:
        paths:
          - pathType: Prefix
            path: /service-a(/|$)(.*)
            backend:
              service:
                name: service-a
                port:
                  number: 8080
          - pathType: Prefix
            path: /service-b(/|$)(.*)
            backend:
              service:
                name: service-b
                port:
                  number: 8080

kubectl apply -f ingress-manifest.yaml

Use port-fording to bind the Ingress controller HTTP port to your localhost and test if the pods are reachable:

# find out what name the controller pod has
kubectl get pods -n ingress-nginx
kubectl port-forward -n ingress-nginx ingress-nginx-controller-<name> 8080:80
curl localhost:8080/service-a
# outputs: pod-a
curl localhost:8080/service-b
# outputs: pod-b

Setting Up the Gateway Version

Let's create a new cluster for the analogue example with the Gateway

kind create cluster --name gateway-example
kubectl config use-context kind-gateway-example

NGINX Kubernetes Gateway

We're going to stay with NGINX, but use the NGINX Gateway implementation next. Let's first install the general Kubernetes Gateway API:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.6.2/standard-install.yaml

Next we install the NGINX Gateway Controller and the GatewayClass.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: k8s-gateway.nginx.org/nginx-gateway-controller

kubectl create namespace nginx-gateway
kubectl create configmap njs-modules --from-file=nginx-gateway/httpmatches.js -n nginx-gateway
kubectl apply -f nginx-gateway/nginx-gateway-controller.yaml
kubectl apply -f gatewayclass-manifest.yaml
# check it's working
kubectl get pods -n nginx-gateway
kubectl get gatewayclasses.gateway.networking.k8s.io

Same Pods and Services

And again our pods and services A and B:

kubectl apply -f pods-service-manifest.yaml

Gateway Instead of Ingress

Instead of deploying the Ingress we will now add the Gateway instead. Thinking in roles intended for this, deploying the Gateway would be the job of a cluster operator. This is a minimal working example that uses the NGINX GatewayClass that we already installed. We define one HTTP Listener with no hostname, so we catch all incoming traffic. You can have multiple Listeners each with a different host:port combination.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: combined-gateway
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      protocol: HTTP
      port: 80

Adding the Routes

We are using HTTPRoutes and attach them to the Gateway with the parentRef. name references the name of the Gateway and sectionName the name of the Listener in that Gateway.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: service-a
spec:
  parentRefs:
    - name: combined-gateway
      sectionName: http
  hostnames:
    - localhost
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /service-a
    # filters:
    #   - type: URLRewrite
    #     urlRewrite:
    #       path:
    #         type: ReplacePrefixMatch
      backendRefs:
        - name: service-a
          port: 8080

# apply gateway and routes
kubectl apply -f gateway-manifest.yaml

The commented section is how Gateways specify the URL rewrite that we know from the Ingress annotation above. It is commented because the URLRewrite filter is currently not supported by the NGINX Gateway. Have I already told you that this NGINX Gateway controller is also a beta version 😀😅? To make the example work anyway I configured the pods to ignore the prefix with which they are called.

# find out what name the controller pod has
kubectl get pod -n nginx-gateway
kubectl port-forward -n nginx-gateway <gateway-name> 8080:80
curl localhost:8080/service-a
# outputs: pod-a
curl localhost:8080/service-b
# outputs: pod-b

Conclusion

You have seen how easy it is to replace Ingress with Gateway while keeping your services and pods. The next step from here is exploring the new load balancing features of Gateways which allow e.g. canary releases and using other types of Routes than just HTTPRoutes. Also make sure to check out what implementations of the Gateway API already exist.

Debugging Tips

Lastly, I want to share some of the debugging tools that I used. As always when you learn new technologies it is important to understand the trial-and-error-loop. While you usually find a lot of information about problems in your Kubernetes cluster in the events, that didn't work for me with Gateways. I started to read out the status fields of the Gateways and HTTPRoutes. They will show you misconfigurations and display e.g. how many routes are attached to a Gateway listener. You find the status when you execute:

kubectl describe gateway
kubectl describe route

K9s

Do you already know k9s? It is a terminal UI for Kubernetes and I use it a lot if there is no Kubernetes Dashboard or other means of management tool. Also very handy for a quick look into a KIND cluster.

Do you need Kubernetes or Go Experts?

We are a Software Agency from Stuttgart, Germany and can support you on your journey to deliver outstanding user experiences and resilient infrastructure. Reach out to us.

Something Else?

Make sure to check out our build tool for large multi-language applications: https://bob.build