Traffic IP Addresses and Traffic IP Groups
In a typical network, your back-end servers will be arranged in a local network, with local IP addresses, and will not directly contactable from the Internet. The front-end Virtual Traffic Manager machines will have externally available IP addresses, and will be able to connect to the back-end machines over the local network.
The front-end machines will have permanent IP addresses on each network, with the front-end addresses visible and routable from the Internet. These IP addresses are configured by the OS, and if the machine fails, the IP address is lost.
For this reason, these permanent IP addresses are not suitable to use when you publish your services. In the event of a hardware or system failure, your services would become partially or wholly unavailable.
The Virtual Traffic Manager’s fault tolerance capability allows you to configure Traffic IP addresses. These IP addresses are not tied to individual machines, and the Virtual Traffic Manager cluster ensures that each IP address is fully available, even if some of the clustered Virtual Traffic Manager machines have failed.
In a typical fault-tolerant configuration, the DNS name used to publish your services is configured to resolve to the traffic IP addresses in your Virtual Traffic Manager cluster. This ensures that your services are always fully available.
The traffic IP addresses are arranged into a Traffic IP group. This group spans some or all of your Virtual Traffic Manager machines. The machines negotiate between them to share out the traffic IP addresses, each Virtual Traffic Manager then raises the IP address (or IP addresses) allocated to it.
Setting up traffic IP groups is described in Traffic IP Groups and Fault Tolerance.
If any Virtual Traffic Manager machine should fail, the other Virtual Traffic Managers in the group detect this. One of them then takes over the failed machine’s traffic IP addresses to ensure that the service is uninterrupted.
By default, fault tolerance uses unicast traffic to distribute health information and to manage traffic to multi-hosted Traffic IP Addresses. If you change your configuration to use multicast traffic, the switches that your Virtual Traffic Managers are connected to must support IGMP snooping, and messages broadcast to the multicast address used by your Virtual Traffic Managers should be forwarded to all Virtual Traffic Managers in the cluster.
When you join a Virtual Traffic Manager to an existing cluster, a number of tests are conducted automatically to ascertain whether broadcast messages are correctly forwarded.
Traffic IP Address Modes
The Virtual Traffic Manager supports several modes:
•Single-hosted: Traffic IP addresses are raised on a single Virtual Traffic Manager in your fault tolerant cluster. If that Virtual Traffic Manager fails, another Virtual Traffic Manager will raise that IP address and start accepting traffic.
•Multi-hosted: Traffic IP addresses are raised on all of the Virtual Traffic Managers in your cluster, using a multicast MAC address that ensures that all incoming traffic is sent to all machines. A custom Linux kernel module is used to evenly distribute the traffic between the working Virtual Traffic Managers.
•Route Health Injection: Traffic IP addresses are raised "privately" (on loopback) by all participating Virtual Traffic Managers, and dynamically advertised into the adjacent routing domain, using either Open Shortest Path First, version 2, (OSPFv2) or Border Gateway Protocol (BGP) as the routing protocol. In response, routers direct traffic to the active Virtual Traffic Manager. See Route Health Injection and the Network.
Enabling Multi-Hosted Traffic IP addresses imposes a performance hit due to the additional packet processing required by the Virtual Traffic Managers in your cluster. Empirical tests indicate that CPU utilization will increase by 25-30% at moderate traffic levels (10,000 requests per second), with a corresponding limit on top-end capacity.
Multi-hosted IP functionality is available on all versions of the Virtual Traffic Manager hardware appliance and virtual appliance. It is not included by default with the Virtual Traffic Manager software variant; however, you can download and install it as an additional kernel module. It is supported on Linux kernels, version 2.6.18 and later. For further information regarding supported versions, see the Ivanti Community Web site at https://forums.ivanti.com/s/?language=en_US.
Example Configurations
These configurations assume that you have two Virtual Traffic Managers in your cluster, but can be extended if your cluster contains three or more Virtual Traffic Managers.
Active-Passive Configuration – Single-Hosted and Route Health Injection Modes
Suppose your Web site’s external DNS name maps to the IP address 162.34.64.29. You have two Virtual Traffic Manager machines handling traffic for a number of back-end Web servers:
•With single-hosted mode, you can set up a single-hosted traffic IP group spanning both Virtual Traffic Manager machines, containing this single IP address. The Virtual Traffic Managers will negotiate and one of them will raise the IP address. It handles all the incoming requests. The second Virtual Traffic Manager machine is available on standby. If the first machine should fail, the second machine takes over the IP address and starts to manage the traffic.
•With Route Health Injection (RHI) mode, you can set up an RHI traffic IP group spanning both Virtual Traffic Managers, containing the single IP address. Set one Virtual Traffic Manager as active and the other as passive. Both Virtual Traffic Managers advertise the IP address, using an OSPFv2 and/or BGP (depending on your choice of routing protocol) metric to express preference (the active Virtual Traffic Manager uses a lower metric). The upstream router (typically your default gateway) chooses the lowest metric route, sending all traffic to the active Virtual Traffic Manager. If the Virtual Traffic Manager detects a failure, it cancels the advertisement. If the router detects a failure, it disregards that route. In either case, the router switches to sending traffic according to the next-best metric advertisement, which in this case is the passive Virtual Traffic Manager.
The advantage of this configuration is that you can be confident that there is sufficient resource in reserve to handle the traffic should one of the two Virtual Traffic Managers fail. Debugging and fault resolution is easier when only one Virtual Traffic Manager is handling traffic.
Active-Active Configuration – Single and Multi-Hosted Modes
In an active-active configuration, both Virtual Traffic Managers manage your traffic. The distribution mode (single-hosted IP or multi-hosted IP) controls how the traffic is shared between them.
With single-hosted mode, you can configure two traffic IP addresses in a traffic IP group, and configure your DNS name to map to the two addresses, such as 162.34.64.29 and 162.34.64.30. The Virtual Traffic Managers will negotiate to raise one traffic IP address each. A DNS server can allocate requests to each IP address in turn (round-robin DNS), and each Virtual Traffic Manager handles the requests it receives.
If one of the machines fails, the other machine will detect this. It will then raise the failed machine’s traffic IP address in addition to its own, and handle all the traffic sent to either address.
With multi-hosted mode, you can continue to operate with one traffic IP address, simplifying your DNS and reducing the number of externally facing IP addresses you require. The traffic IP address is raised on all of the Virtual Traffic Managers in the traffic IP group, and incoming traffic to that IP address is shared evenly between the Virtual Traffic Managers.
Active-Active with Loopback (Single External Address, Single Hosted Mode)
If multi-hosted mode is not available or prohibited in your infrastructure, you can distribute traffic load to all of the Virtual Traffic Managers in your cluster, while still using a single external traffic IP address.
This configuration involves an additional "loopback" virtual server that listens for traffic on the external traffic IP address and then load-balances the requests across the Virtual Traffic Managers using their permanent IP addresses.
First, create your primary virtual server that processes traffic and load-balances it across the back-end servers. Configure this virtual server so that it listens on internal IP addresses and ports on each Virtual Traffic Manager, rather than externally accessible ones. For example, the virtual server could listen on 192.0.2.1:80 and 192.0.2.2:80, where the IP addresses are internally visible only.
Then, create a second “loopback” virtual server that listens on the single external IP address and immediately distributes traffic to the primary virtual server across the various Virtual Traffic Manager machines in your cluster.
As in the active-passive example, set up a single traffic IP address that will be raised by one Virtual Traffic Manager only. Any traffic coming in to this address should then be processed by the simple loopback virtual server, which is listening on that traffic IP address. The loopback virtual server should immediately select a loopback pool that contains the internal IP addresses of the Virtual Traffic Manager machines in the cluster; the loopback pool should use a either round-robin or least connections load balancing to evenly distribute traffic across the Virtual Traffic Manager machines in the cluster. It should not use a load-balancing method that is influenced by response time, as that will give very uneven request distribution.
The loopback virtual server uses little processing power. Ensure that all of the CPU-intensive processing is performed by the primary virtual server - tasks such as SSL decryption, rules, content compression, and so on.
This method splits the load of more intensive tasks between the two Virtual Traffic Managers. If either Virtual Traffic Manager fails, the service will continue to run (perhaps with a momentary interruption to some traffic). For example, if the Virtual Traffic Manager that is hosting the external traffic IP address were to fail, the traffic IP would be transferred to the remaining Virtual Traffic Manager. The loopback pool will detect that one of the nodes was unavailable and direct all traffic to the primary virtual server running on the remaining Virtual Traffic Manager.
The target virtual server will observe the connection originating from the loopback virtual server, not the remote client. Generally, the “X-Forwarded-For” or “X-Client-Cluster-Ip” headers can be used to determine the correct source for the connection, but in the common case where SSL requests are forwarded by the loopback virtual server, you should use the ssl_enhance and ssl_trust_magic settings described in the Preserving IP Addresses with SSL Forwarding section in SSL Encryption
Multiple-Redundant (N+M) Configuration
In the earlier cases, two Virtual Traffic Manager machines are used; if one should fail, a single point of failure is introduced into the system. When running mission-critical services, a higher level of protection can be employed by incorporating several additional Virtual Traffic Manager machines to form a multiple-redundant cluster.
Suppose that in normal operation you want to use N active Virtual Traffic Managers. You would then incorporate M passive Virtual Traffic Managers, where M is the number of machines that could potentially fail without reducing the number of Virtual Traffic Managers in operation.
To achieve this arrangement, you would need N+M front-end machines running the Virtual Traffic Manager. You can create a traffic IP group containing N traffic IP addresses, yet spanning all N+M machines. If a machine in your active group fails, a backup machine from the passive group is brought into active duty to take up the load. If another machine fails, an additional passive Virtual Traffic Manager becomes active, and so on. This is a typical clustering arrangement incorporating multiple layers of redundancy.
Using IP Transparency with a Cluster
Using IP transparency with a cluster of Virtual Traffic Manager machines introduces additional complexity because each server node is configured to route to a single Virtual Traffic Manager IP address. However, any of the Virtual Traffic Manager machines in the cluster may send transparent connections to the server nodes, and the nodes must route each response back via the Virtual Traffic Manager that originated the connection.
Active-Passive Configuration
With a single active Virtual Traffic Manager configuration, this can be achieved using the keeptogether setting in a traffic IP group that uses single-hosted IP addresses.
Create a traffic IP group containing two IP addresses; the front-end IP address for incoming traffic, and a back-end IP address that resides on the server side network. Select the keeptogether option in the traffic IP group.
Configure each back-end server to route all traffic via the back-end IP address you configured in the traffic IP group.
With this configuration, both IP addresses will be hosted on the same Virtual Traffic Manager machine. If that machine were to fail, both IP addresses would be migrated to the same passive machine, making it the new active machine. The back-end servers will now route traffic back via the new active machine.
Active-Active Configuration
With a configuration involving multiple active Virtual Traffic Managers, it is necessary to partition your back-end servers into groups, one for each active Virtual Traffic Manager machine. These groups should then be defined as pools within the Virtual Traffic Manager Admin UI, adding each back-end server as a node accordingly.
All servers in the same pool should have their default route configured to be the back-end IP address of the corresponding Virtual Traffic Manager. Please refer to your operating system documentation for more details about how to manipulate your server route settings.
TrafficScript rules can then be used to select the correct pool to route traffic to, based on either of the following items:
•The name of the Virtual Traffic Manager that is managing that connection.
•The local client-side IP address (if using several "keeptogether" Traffic IP Groups in single-hosted mode).
The following code snippet demonstrates how this might work (using the Virtual Traffic Manager name as the selection criteria):
$hostname = sys.hostname();
if( $hostname == "TM1" ) {
pool.use( "TM1_nodes" );
}
if( $hostname == "TM2" ) {
pool.use( "TM2_nodes" );
}
if( $hostname == "TM3" ) {
pool.use( "TM3_nodes" );
}
This configuration does, however, include the limitation whereby if a Virtual Traffic Manager fails the associated pool will become redundant. Additionally, session persistence cannot reliably be used (particularly if multi-hosted IP addresses are in use).
IP transparency can be used selectively. For example, suppose that a Virtual Traffic Manager cluster is managing high volume Web traffic to www.mysite.com, and low volume SMTP traffic to mail.mysite.com. Only the SMTP traffic needs to be transparent. In this case, the following is true:
•www.mysite.com can resolve to several IP addresses in an Active-Active TrafficCluster configuration without IP transparency.
•mail.mysite.com can resolve to a single IP address using the Active-Passive keeptogether configuration described above.
Route Health Injection and the Network
When using Route Health Injection (RHI), the Virtual Traffic Manager communicates with routers in the adjacent routing domain. Once it has established communication, the Virtual Traffic Manager joins the routing domain and advertises RHI traffic IP addresses into it.
Such advertisements are dynamic and respond automatically to your Virtual Traffic Manager configuration changes that create, destroy, or move RHI traffic IP addresses. The advertisements also respond automatically to failures detected by the Virtual Traffic Manager, and through the routing domain's dynamic routing protocols, to network failures that are not local to the Virtual Traffic Manager.
RHI is therefore able to work in a wide variety of deployments. For example, on a small scale to manage simple local failover (such as within a single datacenter rack), and on a large scale to manage traffic distribution and failover between different datacenters within an enterprise or across the whole Internet.
RHI operates using RHI-designated IPv4 traffic IP groups. In a single location, you can use an RHI traffic IP group serviced by either a single active Virtual Traffic Manager, or an active-passive pair of Virtual Traffic Managers (See "Active-Passive Configuration – Single-Hosted and Route Health Injection Modes" on page 43).
RHI does not support Traffic IP groups based on IPv6 addresses.
To increase scale, you can repeat this pattern in further Virtual Traffic Manager datacenter locations as necessary. Different locations use different RHI traffic IP groups, where the Traffic IP addresses in each group are identical, but the OSPFv2 or BGP metrics used are typically different.
For OSPFv2, this scenario requires all datacenters to be in the same routing domain. For BGP, datacenters can be internal or external to the routing domain.
Locations might have different priorities. For example, with a two-location primary-standby datacenter deployment, you configure the following:
1.In the primary datacenter, configure a primary RHI traffic IP group serviced by an active-passive Virtual Traffic Manager pair.
2.In the standby datacenter, configure a standby RHI traffic IP group serviced by an active-passive Virtual Traffic Manager pair.
3.Configure the standby RHI traffic IP group with a very high metric, such that the primary datacenter is always preferred unless it is unavailable.
Alternatively, your datacenter locations might have similar priority, resulting in multiple active locations where routing decisions are based on using the best route, according to the network topology. This is often referred to as an anycast configuration. To achieve it, configure the RHI traffic IP groups in each location with the same metrics.
RHI implemented with BGP over multiple locations requires other parts of your infrastructure to be configured to support it. In other words, you must ensure that the routers between your datacenter locations respect the supplied metrics, and do not have other policies configured that might influence location choice.
The Credentials Used for RHI Communications
To implement RHI, each Virtual Traffic Manager communicates with adjacent routers using OSPFv2 or by establishing a session with BGP. You must configure your Virtual Traffic Manager with suitable credentials to enable the Virtual Traffic Manager to establish communications with an adjacent OSPFv2 or BGP enabled router. This process is called “peering” with the router.
OSPFv2 communication requires multicast.
You can create clusters of Virtual Traffic Managers that use the same credentials. Each cluster (or each Configuration Location within a Multi-Site Manager cluster; see "Multi-Site Cluster Management" on page 404) uses one set of credentials, and therefore all Virtual Traffic Managers in the cluster (or Configuration Location) join the same area (for OSPFv2) or Autonomous System (AS) (for BGP).
If you have multiple locations that require different credentials, the Virtual Traffic Managers in the different locations do not need to be clustered. If, however, you want to cluster them, you must use the Virtual Traffic Manager's Multi-Site Manager functionality.
To enter your OSPFv2 or BGP credentials, use the System > Fault Tolerance page.
For further details about OSPFv2/BGP configuration and RHI traffic IP group configuration, see Traffic IP Groups and Fault Tolerance
Troubleshooting RHI Issues
The Virtual Traffic Manager uses third party routing software for RHI operations. This routing software logs RHI events to a specific file in your file system: $ZEUSHOME/log/routing_sw
The contents of this file are included in your Technical Support Reports (see Getting Help) and can be useful to your support provider in troubleshooting RHI communication problems in your Virtual Traffic Manager deployment.
This log file is subject to the following rotation policy:
•For Virtual Appliance and Cloud instances, the Virtual Traffic Manager performs log rotation automatically using a fixed file size (50 MB by default). Archived logs are compressed with the "gzip" compression method and stored under a name containing the date and a sequential number.
•For software instances, you must enact your own rotation policy. To inform the Virtual Traffic Manager that the log file has been rotated (moved), use the script $ZEUSHOME/zxtm/zebos/bin/reload_logfile post-rotation to restart the logging process in a new file.
An Introduction to OSPFv2 and BGP
The Virtual Traffic Manager supports Open Shortest Path First version 2 (OSPFv2) and Border Gateway Protocol (BGP) as routing protocols for RHI.
OSPFv2 is an “interior gateway protocol”, typically used to distribute routes inside a single Autonomous System (AS) on a network, for example, with ISPs or large company networks. OSPFv2 enables routers to auto-discover and synchronize with other OSPFv2-configured routers in the same AS.
OSPFv2 works at Layer 3, using raw IP packets rather than over TCP or UDP, using a Time-To-Live (TTL) value of 1. It uses multicast addressing to flood routing information to the next router in the chain, and it is able to handle its own error detection and correction. OSPFv2 is typically internal to a body such as an ISP and is quick to converge (when a route changes, convergence is achieved when all routers in a network agree the new quickest route).
For further information on OSPF, see http://en.wikipedia.org/wiki/Open_Shortest_Path_First.
BGP is, by comparison, an exterior gateway protocol, typically used to distribute routing and reachability information outside of individual AS's.
BGP can still be used as an interior gateway protocol. For this purpose, the Virtual Traffic Manager uses Internal BGP (known as iBGP) for information exchange. For communication between routers in different ASs, the Virtual Traffic Manager uses External BGP (known as eBGP).
Unlike OSPFv2, rather than being able to auto-discover their peers, BGP enabled routers require you to define the explicit configuration of the neighbors with which they expect to establish sessions.
For further information on BGP, see www.bgp4.as.
You can configure the Virtual Traffic Manager to use either, or both, of these protocols to advertise IP addresses and to have these advertisements propagate through the network. To enable and use Route Health Injection with your services, see Creating a Traffic IP Group.