RabbitMQ Queues

The RabbitMQ management interface provides data to help monitor queues.

Monitoring Queues

The Queues tab on the RabbitMQ management interface displays the following information when Cherwell Service Host is started:

Information	Description
Name	The name of the queue is a reflection of the service it supports.
Consumers	Each queue must have at least one consumer when it is initially set up.
Messages	Messages are the number of messages currently being held by the queue.

Queue Naming Conventions

Queues names are created using the following pattern:

                            [Code Namespace].messages.[Queue Name].[Partition Key].[ServiceHostIdentifier]

The last three components are described in the following table:

Component	Description
Queue Name	Lists what type of items are in the queue and is used to discern how to interpret metrics that deal with the number of items in the queue.
Partition Key	Primarily used for scenarios where multiple CSM instances share a single RabbitMQ instance. For on-premises environments, the partition key should be constant, but may get changed on upgrade of CSM.
Service Host Identifier	Allows you to determine machine-specific queues where applicable. Not all queue names are scoped per machine.

Consider this queue name as an example:

Trebuchet.Plugin.core.services.messages.Starttaskmessage. 909e287f-a1fa-4f53-be17-194a0a061770.15424

This queue name has the following components:

Queue name: Starttaskmessage
Partition Key: 909e287f-a1fa-4f53-be17-194a0a061770
Host identifier: 15424

Queue names can indicate information about their ready and unacknowledged (unacked) message volumes. In the following table, some of these elements are described. Large queues refer to the total number of items in the queue:

Queue Name Element	Description
Sendmailrequest	Large queues are representative of items pending retrieval from the database for transmission.
Automatedprocessmsg	Large queues are representative of items which are pending processing for AP (Automated Processing) Events. Note: Each message does not usually correlate to a single Business Object as a message can signal to process multiple records.
Schedulemsg	Large queues are representative of items that are pending processing by the scheduler. Each message includes items that are due to be run but not items that have not reached time to execute.
Starttaskmessage	A request to create a new worker for scaling. The number of messages waiting should always be small (less than 20% of the maximum number of workers).
Stoptaskmessage	A request to remove a worker. This occurs when the Cherwell Service Host is scaling down workers after high activity. The number of messages waiting should always be small (less than 20% of the maximum number of workers).
Restarttaskmessage	Each worker sends a heartbeat periodically. A process monitors workers, and if the worker has a critical error, the worker will be restarted. This message is used to queue a worker to be shutdown. Note: Any messages indicate critical failures for the service host and should be investigated.
Emaildeliverymessage	Large queues are representative of items pending transmission.
Heartbeatmessage	Each worker generates periodic messages to alert the Cherwell Service Host that the worker is responsive and report metrics. Large message counts could indicate a failure in the Cherwell Service Host monitoring. (This should be very close to 0.)

Note: Select an item on the RabbitMQ management interface to drill down and investigate a specific queue.

Exchange Types

Queues names are created and managed by CherwellMQS. They are based on the type of exchange the queue is bound to. CSM uses two types of exchanges:

Direct: Delivers a message to a single queue based on the message routing key.
Fanout: Delivers messages to multiple unique queues bound to the exchange. In a Fanout exchange, the routing key is ignored.

The name for a queue that is bound to a Direct exchange is a combination of the name of the message, the tenant's partition key, and depending on the behavior of the message, the instance ID. The name for a queue that is bound to a Fanout exchange includes the message handler's target name to create multiple queues to handle multiple messages per instance.

Note: Since queue names created automatically by CherwellMQS with specific meaning, Users should not attempt to edit or rename queues.

Analyzing Messages

The Overview tab on the RabbitMQ management interface displays data that helps determine the health of the queue over time. Monitor trends of published messages, consumed messages, and queued messages. Pay attention to the following graphs in the Overview tab:

Queued messages: This graph displays the number of messages queued over time. If there is a growth rate, then the queue will fall behind and eventually overflow.
Message rates: This graph displays the most basic and telling metric in the queuing system. If the number of messages being Published is greater then the number of Consumer acknowledged (acked) messages, then the system is getting more work requests then it can handle. In this case, steps must be taken to prevent a failure

Analyzing Queuing Data

Decisions you make regarding scaling CSM services or multiplying the number of workers available on any single Cherwell Service Host machine depend on a number of factors. Each organization has unique needs, and you must consider the needs of your organization to determine the best scaling options.

For example, you may see short-terms spikes in the amount of queued work based on business demands or on your organization's business hours. Or, you may notice one CSM microservice is continually filling the queue when it is enabled on a machine where other microservices are enabled.

When to scale or distribute microservices:

Consider adding another machine instance of the Cherwell Service Host in these cases:

The numbers of consumers, pending messages, or messages queued grows or remains even over a period of time.
The CPU of the Cherwell Service Host machine consistently hits or exceeds unacceptable ranges.

Use the Server Installer to add the Cherwell Service Host to additional machines. See Advanced Configurations for the Cherwell Service Host.

When to multiply the number of workers on a Cherwell Service Host machine:

Consider multiplying the number of workers on a single machine in these cases:

The numbers of consumers, pending messages, or messages queued grows or remains even over a period of time.
The CPU of the Cherwell Service Host machine is not fully used.

Use the Configuration Command-Line Utility to increase the number of workers per virtual processor on each Cherwell Service Host machine. See CherwellMQS Command Line Options.