Using the Traffic Manager as a Kubernetes Ingress Controller

This feature is applicable only to version 22.3 and later.

The Ingress Controller is a Traffic Manager feature that implements and meets the requirements of a Kubernetes Ingress Controller.

This feature enables you to position your Traffic Manager as an ingress point for your Kubernetes infrastructure, and to control your Traffic Manager configuration through standard Kubernetes mechanisms.

This guide assumes you have an understanding of Kubernetes features and capabilities, and that you have a previously-configured Kubernetes service infrastructure with which you want to add the Traffic Manager, In particular, make sure you are familiar with the following Kubernetes concepts:

•Running services on a Kubernetes cluster

•The Ingress resource concept and specification

•The IngressClass specification

This feature was previously made available as part of an optional external solution. The Kubernetes resource specification for ingress supported by the former Ingress Controller feature has changed in Kubernetes v1.18.

Supported Features

The Traffic Manager acting as an Ingress Controller includes support for the following functionality:

•Handling of HTTP and HTTPS traffic, including HTTPS offload

•Let’s Encrypt certificate generation and renewal

•The ability to watch either a single, specified namespace, or all namespaces except "kube-system"

•Session persistence

•Health monitoring

•Integrated logging with the Traffic Manager event log

High Availability

The Traffic Manager can be deployed and configured as a cluster of instances for high-availability (HA) and redundancy. With the Ingress Controller feature enabled across the cluster, each Traffic Manager instance can assume the role of primary node. However, only the Ingress Controller on the primary node will attempt to apply configuration to the cluster. The other Traffic Manager instances still see changes on the Kubernetes platform so as to keep their state up-to-date, but they will not attempt to apply any configuration to the cluster unless they then become the primary instance.

Logging

Log messages concerning Ingress Controller activity appear in the Traffic Manager Event Log, and are prefixed with "ingresscontroller:”.

With verbose logging disabled (default), only errors are reported in the logs. If additional logging information is required, enable the vtm_verbose setting on the System > Ingress Controller page. For more details, see Configuring the Traffic Manager as an Ingress Controller.

Scope and Restrictions

The Traffic Manager remains external to the Kubernetes cluster, and is configured automatically through interaction with the Kubernetes Ingress resource.

As demonstrated in this diagram, the Traffic Manager's Ingress Controller function watches the configured Kubernetes API for Ingress and IngressClass resources. Based on these, the Traffic Manager creates resources and service configuration within your Traffic Manager cluster to meet the needs of the Kubernetes-based services.

As well as monitoring the Kubernetes API for Ingress resources, the Traffic Manager also monitors Secret resources, and creates SSL server certificate entries for any Secret resource of the type kubernetes.io/tls, assuming they also match any label selector specified in the configuration.

You can operate a cluster of Traffic Manager in this role for high-availability and redundancy purposes (see High Availability). However, the Traffic Manager is not able to load-balance requests directly to the Kubernetes pods. This is because the pods have an IP address only on the Kubernetes internal network, to which an external Traffic Manager cannot access. The data path to the pods must therefore be via a NodePort-based Service (see https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types).

Session persistence and health monitoring function only if the identity of a pod can be inferred from another piece of information known to the Traffic Manager; in this case, the Kubernetes compute node on which the pod is running. Accordingly, these features require that a podAntiAffinity rule (see https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) is defined in the Deployment (or ReplicaSet) specification to avoid multiple pods from the same Deployment being scheduled to the same compute node. Furthermore, these features require all defined Service objects within the Ingress to specify the spec.externalTrafficPolicy: Local setting, so as to avoid kube-proxy re-routing requests to a different compute node (see https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport).

Defining the Traffic Manager as the Controller in Your Ingress Specification

The Traffic Manager supports the standard Kubernetes Ingress specification, as well as some custom metadata annotations (see Ingress Metadata Annotation Reference).

For the Traffic Manager to recognize Ingress and IngressClass resource as applicable to this function, such resources must use an IngressClass defined with the following field:

spec:

controller: pulsesecure.net/vtm-ingress-controller

While deprecated, the Traffic Manager will also act on Ingress objects that contain the value “vtm” in the kubernetes.io/ingress.class annotation.

IP Address Support

The Traffic Manager Ingress Controller function supports two types of user-supplied IP address through the following Ingress annotations:

•vtm.pulsesecure.net/ip-address: This annotation should be used when a dedicated IP address is allocated to an Ingress. The full range of functionality provided by other Ingress annotations can be used with this option.

•vtm.pulsesecure.net/shared-ip-address This annotation enables a single IP address to be shared by multiple Ingress objects. These can use either the same or different ports; when the same port is specified by multiple Ingresses, the normal Ingress routing rules are applied in the order the Ingress objects were created. The following restrictions apply to shared IP services:

• ingress.kubernetes.io/ssl-passthrough cannot be used, since the HTTP payload needs to be read to perform request routing.

• vtm.pulsesecure.net/ssl-redirect can only be used if the primary Ingress bind port is 443. This can also not be specified if an existing Ingress sharing the IP uses port 80 as its bind port. If this is specified, no subsequently-created Ingress object can use port 80 as its bind port.

•vtm.pulsesecure.net/use-lets-encrypt cannot be specified if an existing Ingress sharing the IP address uses port 80 as its bind port. If this annotation is specified, no subsequently-created Ingress object can use port 80 as its bind port.

If neither of the ip-address annotations are specified, the Traffic Manager will attempt to use a specified default Traffic IP Group. If no default is specified, an error is reported.

To learn more about applicable Ingress resource annotations, see Ingress Metadata Annotation Reference.

Ingress Spec Field Support

An Ingress resource definition includes a spec (specification) field to define the desired state and rules to which incoming requests must match. The Traffic Manager supports the following additional definitions within the spec: field:

•TLS: If specifying a TLS section in the Ingress for SSL offload, it is important that the relevant Secret resources are created beforehand (unless Let’s Encrypt support has been specified; see Let's Encrypt Support). If an attempt is made to apply an Ingress before the Secrets exist, this will cause an error on the Traffic Manager and the Ingress will not be applied successfully.

•Backend: If this section is specified, the relevant Traffic Manager pool will be created and set as the default pool for the virtual server. This section can be used alone or in conjunction with a spec.Rules section.

•Rules: If this section is specified, a separate vTM pool will be created for each required backend. A TrafficScript rule will be applied to the virtual server to route requests to the relevant pool. The values of path and host fields support wildcard matches.

Let's Encrypt Support

If, in the Ingress resource definition, the vtm.pulsesecure.net/use-lets-encrypt custom metadata annotation is set to “true”, the behavior of the Traffic Manager changes with regards to how the "spec.TLS" section (see Ingress Spec Field Support) is processed.

Instead of expecting a Secret with the specified name to already exist, the Traffic Manager attempts to obtain a new certificate for the specified hostnames from Let’s Encrypt (or another ACME service), and creates a new Secret containing the certificate. A separate part of the Ingress Controller function monitors all Secrets created by the Traffic Manager and renews the certificates eight days before they expire. For Let’s Encrypt to issue a certificate, it must successfully challenge every domain name in the request. It is therefore necessary to ensure DNS can resolve the domain names to the listen IP address of the Ingress before the Ingress is created.

Prerequisites

Before using this feature, make sure you meet the following prerequisites:

For Kubernetes:

•The minimum supported version is v1.19. An error message is logged in the Traffic Manager event log if the Kubernetes cluster version the ingress controller has connected to is not appropriate.

•A fully deployed Kubernetes cluster.

•A Service Account with appropriate permissions granted for reading and listing IngressClass resources in a Kubernetes namespace of interest.

•Kubernetes Service resources of type NodePort

•See also Scope and Restrictions

For the Traffic Manager:

•Version 22.3 or later.

•A fully deployed Traffic Manager instance or HA cluster.

•Kubernetes nodes should be accessible to the Traffic Manager through your network.

Preparation for Use

To use this feature, perform the following steps:

•In Kubernetes, create ServiceAccount or kube config, as well as IngressClass and Ingress resources according to the Kubernetes documentation.

•In the Traffic Manager, configure and enable the Ingress Controller functionality.

Configuring the Traffic Manager as an Ingress Controller

To configure the Traffic Manager as an Ingress Controller for your Kubernetes-based services, perform the following steps:

1.Make sure you have access to a chosen Kubernetes cluster

2.Define a ServiceAccount with sufficient rights

3.Retrieve your Kubernetes access token string from a Secret created for the ServiceAccount

4.Log in to the Traffic Manager Admin UI and access the System > Ingress Controller page

The following configuration page appears:

5.Configure the following settings:

•enable_ingctrl: Enables the Ingress Controller feature on the Traffic Manager

•k8s_config_path: (Optional) The path to a Kubernetes user configuration file (for example, "/etc/kubernetes/admin.conf"). If this field is not specified, the controller uses the values configured in k8s_endpoint and k8s_token to connect to the API server.

•k8s_endpoint: The Kubernetes API endpoint (for example, "https://api.mykube.local:6443"). You must specify this setting, even if a Kubernetes configuration file is specified in k8s_config_path.

•k8s_token: The Kubernetes API authentication token. You must specify this setting, even if a Kubernetes configuration file is specified in k8s_config_path. If a configuration file is also specified, this token must have permission to query "service" and "endpoints" resources for all namespaces specified in the configuration file. If a configuration file is not specified, this token must have permission to read Ingress resources and read/write Secret resources in all relevant namespaces.

•k8s_verify_ssl: Whether to verify the SSL certificate of the Kubernetes API server. This setting is used by both the Ingress Controller and any Service Discovery pools on the Traffic Manager.

•vtm_password: The password of the Traffic Manager user authorized to make REST calls to the Kubernetes API server.

•vtm_verify_ssl_cert: Whether to verify the SSL certificate of other Traffic Managers in your HA cluster.

•vtm_verbose: Enable verbose logging to the Traffic Manager event log.

•vtm_enable_customizations: Whether to allow the automatically-generated Traffic Manager configuration to be overridden by user-specified customizations. If this option is enabled, the vtm.pulsesecure.net/pool-customization and vtm.pulsesecure.net/virtualserver-customization annotations are read. If this option is disabled, any customizations are ignored.

•vtm_enable_trafficscript: Whether to permit custom TrafficScript rules to be applied to the automatically-generated virtual servers. If this option is enabled, the vtm.pulsesecure.net/request-trafficscript and vtm.pulsesecure.net/response-trafficscript annotations are read. If this option is disabled, annotations are ignored.

6.To apply these settings, select Update.

Ingress Metadata Annotation Reference

This section lists all metadata annotations to the Ingress resource that control behavior of the Ingress Controller functionality in the Traffic Manager:

{"properties": {    
    "basic": {
        "type":"http",
        "timeout":5,
        "use_ssl":true
    },
    "http": {
        "status_regex":"^403$"
    } 
}}

{"properties": {
    "basic": {         
        "type":"cookie",
        "cookie":"ChocChip"     
    } 
}}

{"properties": {     
    "basic":{         
        "passive_monitoring": false
    } 
}} 

http.addHeader("X-Via", "Pulse vTM");

if(http.getResponseCode() == 500) {
   http.redirect("https://my.services/sorry") ; 
}

{"properties": {
    "web_cache": {
        "enabled": true
    } 
}}