Metering and Monitoring the Services Director
Usage Metering and Activity Metrics (CSP Customers Only)
The Services Director automatically meters usage on a regular basis, and it optionally sends this information to Ivanti for billing purposes. By default, it records this information once per hour.
If a Traffic Manager instance is active, the Services Director polls it to obtain total throughput and peak activity metrics. The Services Director creates a metrics log file with one line of metrics data for each Traffic Manager instance. Each line of metrics data records the name of the instance, the time elapsed since the resource was created, and the polled metrics. If an instance is not active, only the elapsed time is recorded.
If you want to generate usage or billing information, typically you process all metering log files and aggregate the results. You should use caution when aggregating data results for billing since metering records include failed deployments.
Generating log files has a cumulative impact on disk space.
The Services Director collects metering data from Traffic Manager instances as follows:
•Instances that are at version 9.4 or earlier (or have no REST API enabled) have their metering collected through SNMP.
•Instances that are at version 9.5 or later with the REST API enabled have their metering collected through their REST API. If REST-based metering fails (or is not possible), the Services Director falls back to collecting using SNMP. Any metering issues will be included in the warning logs, as before.
The Services Director records the most recent metrics information for each instance in the inventory database. You can obtain this data using the REST API. The REST API does not supply bulk metrics data.
The Metering Log file is structured as follows:
•The first row contains version data for the metering log format. This first line can be ignored by customers. Ignore this line when you aggregate data for billing.
•Each subsequent row records one set of metrics for a Traffic Manager instance, in comma-separated value (CSV) format.
•The final line contains an MD5 hash of the previous lines. Ignore this line when you aggregate data for billing.
Each line of metrics contains the following fields:
|
Field |
Description |
|---|---|
|
Timestamp |
The date and time, in UTC format, that the line was written. |
|
Instance ID |
The unique instance ID for the Traffic Manager instance. |
|
Instance Tag |
This information may be empty but it is included, even if empty. |
|
Owner |
Optionally, the owner of the Traffic Manager instance. |
|
Cluster ID |
The cluster for the Traffic Manager instance. |
|
Management IP |
The management IP address of the Traffic Manager instance. |
|
Instance SKU |
The SKU (or SKU combination) assigned to the Traffic Manager instance (at the time of writing to the log). The SKU might vary between readings, and variations are not recorded in the metrics log file. This property includes a hash of features applicable to the SKU. Ignore these features for billing purposes. |
|
Feature Pack |
The feature pack assigned to the Traffic Manager instance (at the time of writing to the log). |
|
Deploy Time |
The length of time (in days, hours and minutes) since the instance was deployed. |
|
Throughput |
The number of bytes sent by the Traffic Manager instance, as recorded in the SNMP counter. This number is cumulative and is reset whenever the Traffic Manager instance is restarted. It is not the throughput since the latest metering action. To generate usage or billing information based on throughput, you should set your aggregating script to detect a drop in throughput and designate this as a restart. This property is applicable to active Traffic Manager instances only. For Idle or Inactive instances, it contains a value of 0 (zero) in the log. For uncontactable instances, it contains a value of -1 in the log. |
|
Peak Throughput |
The highest number of bytes sent by the Traffic Manager instance in any second of the previous hour. This property is applicable to active Traffic Manager instances only. For Idle or Inactive instances, it contains a value of 0 (zero) in the log. For uncontactable instances, it contains a value of -1 in the log. |
|
Peak Requests |
The highest number of requests received by the Traffic Manager instance in any second of the previous hour. This property is applicable to active Traffic Manager instances only. For Idle or Inactive instances, it contains a value of 0 (zero) in the log. For uncontactable instances, it contains a value of -1 in the log. |
|
Peak SSL Requests |
The highest number of Secure Socket Layer (SSL) requests received by the Traffic Manager instance in any second of the previous hour. This property is applicable to active Traffic Manager instances only. For Idle or Inactive instances, it contains a value of 0 (zero) in the log. For uncontactable instances, it contains a value of -1 in the log. |
|
Instance Bandwidth |
The bandwidth (in Mbps) allocated to the Traffic Manager instance. |
|
Record Hash |
An MD5 or similar hash of the record from the Services Director license file for tamper detection. Ignore this for billing purposes. |
If metrics are not collected for a period of time, peaks for the missing time are not recorded. If you reduce the metering interval, the peak values are still relative to the previous hour rather than the time since metrics were last collected.
Creating Metering Logs (CSP Customers Only)
By default, the Services Director retains and archives metering log files in the /var/log/ssc/ metering directory. You define the log output location when you first configure the Services Director.
To allow the Services Director to send metering logs to Ivanti, enable the phone home feature through the REST API or Services Director VA. See the Pulse Secure Services Director Getting Started Guide.
The metering phone home service requires DNS to find the phone home server. If you do not have DNS enabled on your Services Director, add the phone home IP address (currently 67.199.253.230), to the VA's static hosts mapping.
To enable the phone home feature through the REST API, perform a PUT request on the settings/phone_home resource, setting the phone_home_enabled property to true. This property is set to false by default.
A warning e-mail will be sent every 24 hours if phone-home is enabled and Services Director is unable to connect to the phone-home server.
If the phone home feature is disabled, you must manually extract the metering log files and send them to Ivanti. Employing phone home on your system ensures that the process is entirely automatic, secure, and without staffing overhead.
The Services Director employs a cron job, created during installation, to run a script that prepares a metering data archive file in this format:
<sd_hostname><controller_license_name>.zip
The script sends this file to a secure SFTP server at Ivanti. The cron job runs once a month, at a randomly selected date and time. The Services Director authenticates the destination SFTP server against a pre-configured locally stored host key. You can change this key by placing a new value in:
~/.ssh/known_hosts
If the host key is not found, or does not match the SFTP server, the phone home process stops.
The Services Director makes two attempts to send the log file. If the first attempt fails, the Services Director waits a random period of up to 12 hours and then tries again. If the second attempt fails, the Services Director sends an alert to the email notification list. Equally, if the file is successfully sent, the Services Director sends an alert to the email notification list announcing the success.
Errors and warnings pertaining to the phone home mechanism are all logged to the file metering_phone_home.log. This file provides a source of debugging information for problem resolution.
You can also extract metering logs using the Services Director VA. See the Pulse Secure Services Director Getting Started Guide.
Metering Settings
You can view and modify various metering settings Services Director in the /api/tmcm/2.9/settings/metering resource. These settings do not normally need to be modified from their default values. For details, see Using the ServURI Root Partsices Director REST API.
You set metering settings for the Services Director as follows:
•alerts_and_notifications - A Boolean that controls how metering alerts and notifications are handled. The default value is true. This indicates that the metering health icon will be displayed in the header for the Services Director VA graphical user interface, and that daily emails about metering warnings will be sent.
•meter_interval - The period of time, in seconds, between metering actions. The range is from 1-3600. The default value is 3600 seconds (1 hour).
•snmp_enabled - A Boolean that controls whether SNMP is used. SNMP is used to gather certain types of information (such as metering) from the Traffic Managers in the estate of the Services Director. The default value is true.
•log_check_interval - The period of time, in seconds, between checks for log space. The range is from 1-3600. The default value is 3600 seconds (1 hour).
Creating Metering Records Manually
The Services Director includes a command line utility, prepare_metering_records , to manually prepare metering records for processing. You can use this script to create the metering archive file ready for transmission to Ivanti.
To create metering records
•At the system prompt, enter:
prepare_metering_records [--help] [--force]
--force - Runs the script without prompting for user input.
Creating Metering Records Using the Phone Home Script
You can run the metering_phone_home script to manually perform a phone home operation. This script is stored in the bin directory of the Services Director installation directory. For example, in:
INSTALLROOT/bin
To run the metering phone home script, at the system prompt, enter:
metering_phone_home -v/--verbose -n/--noretry
-v/--verbose - Redirects logging to stdout instead of a file.
-n/--noretry - Prevents further attempts at sending.
Health and Performance Monitoring
Each Services Director in your deployment monitors the health of all other peer Services Directors and deployed Traffic Manager instances recorded in the inventory database. If a failure occurs, the Services Director records a warning entry in the event log and sends an email notification to any system administrators declared in the database.
You can configure the "From" e-mail address of alert e-mails. This address can be set in INSTALLROOT/conf/email_config.txt, in the common section, as from_address. The symbol "$fqdn" will be replaced by the fully-qualified domain name of the instance host, or an IP address where Universal Licensing is used. The other sections in this file should not normally be modified. For Services Director installs on AWS it is likely that you will need to change this setting to be an address that is resolvable to the instance's public IP.
The Services Director also monitors supported versions of deployed Traffic Manager instances for a number of key performance metrics. You can obtain these metrics through the REST API monitoring resource.
A series of inter-controller REST API requests are used to identify the status of each Services Director and Traffic Manager instance in your deployment. You must make sure that each Services Director has network access to all other Services Directors and instance hosts. You must also enable the REST service on your running Traffic Manager instances and record the REST access details in the inventory database.
This process is performed automatically for Traffic Manager instances that are deployed by the Services Director. However, you must enable and record REST API credentials for externally-deployed instances manually to allow monitoring to take place for these instances.
Your Services Directors in an Active state are monitored.
Monitoring Settings
By default, all active Services Directors share the task of monitoring. You can alter this behavior by modifying the Services Director's monitoring mode settings in the REST API manager resource. These settings do not normally need to be modified from their default values. For details, see Using the ServURI Root Partsices Director REST API.
You select whether each Services Director individually monitors all other Services Directors and Traffic Manager instances in the deployment, shares the responsibility of monitoring a proportion of Traffic Manager instances with other Services Directors, or performs no monitoring actions at all.
You can view and modify various monitoring interval settings for the Services Director in the REST API monitoring resource. These settings do not normally need to be modified from their default values. For details, see Using the ServURI Root Partsices Director REST API.
You set distinct interval values for monitoring the Services Directoras follows:
•Monitoring Interval - The period of time, in seconds, that must elapse between health checks. The default value is 60. A setting of 0 forces the Services Director to use a predefined short interval suitable for deployments that require very frequent monitoring.
•Failure Identification Interval - The period of time, in seconds, that must elapse between continuous health check failures before your Services Director determines that a service failure has occurred. This setting helps to prevent against transient network errors being incorrectly identified as service outages. The default value is 180.
•Overdue Monitoring Warning Interval - The period of time, in seconds, that must elapse before any pending monitoring actions are considered overdue. This might occur during periods of unusually heavy load. If defined, a breach of this interval causes the Services Director to issue an alert. The default value is 300.
•Warning Email Interval - The period of time, in seconds, that must elapse before subsequent email alerts are sent. You can use this setting to avoid large numbers of emails being sent, one for each occasion a warning is triggered. A new email is sent only after this interval has passed. This email contains all monitoring events since the previous email was sent.
Retrieving Monitoring Data
You can retrieve stored monitoring state data from the Services Director by using the REST API monitoring resource. This resource is read-only and supports only the REST API GET request method. For details, see Using the ServURI Root Partsices Director REST API.
You can access the following child elements through the REST API monitoring resource:
•/monitoring/manager - This element contains monitoring state data for all of your Services Directors, whether or not they have failed.
•/monitoring/host - This element contains monitoring state data for all of your service hosts, whether or not they have failed.
•/monitoring/instance - This element contains monitoring state data and key performance metrics for your Traffic Manager instances, whether or not they have failed.
•/monitoring/failures - This element contains a pair of arrays for failed Services Directors and Traffic Manager instances. You can use this element to retrieve a list of currently failed devices without needing to check the status of each one individually.