Best Practices for Performance

For best results, periodically review performance best practices to analyze and troubleshoot performance issues.

Advice on performance is also available for the following specific areas:

Business Objects (see Business Object Performance).
Automation Processes (see Performance Considerations for Automation Processes).
Cherwell REST API (see Cherwell REST API Performance).
Relationships (see Relationships and Performance).
Form Arrangement tabs (see Define General Tab Properties for a Form Arrangement).
Widgets (see Add a Widget to a Dashboard (General)).

Determine the Affected Location

Investigate if the entire system or a specific area of the system is perceived as slow. Specific areas might include:

CSM Desktop Client
CSM Administrator (example: Blueprint publishing)
CSM Browser Client
CSM Portal
Certain Business Objects (example: Incident, Change Request)
Dashboard content
One-Step™ Actions
Internal databases (example: CMDB)
External databases (example: Active Directory, SCCM)
External tables

In addition, determine if the slowness is perceived in one of the following environments:

Geographic location: Offices in other cities, states, or countries.
Remote employees: Employees who are connecting via wireless, using a VPN, or working from home.
Hosted or on-premises: Installation method of the affected environment.

Evaluate the Installation Configuration

Ideally, Cherwell server components should run on a dedicated machine. However, some customers may co-locate CSM with other applications or utilities. In this case, you must consider these additional applications or utilities when assessing performance.

Investigate the installation configuration of the affected environment, which affects how you analyze performance and who implements the recommendations. After connecting to the environment (either via the hosted server or via a remote session), consider the following:

Load balancing.
Available memory.
Available disk space.
Subscription numbers.
Number of processors.
Other installed applications.
Separate installed applications.
Distribution of servers (on one machine or several).
Available Central Processing Units (CPUs).
Note: If you are using a virtual machine, ensure that all memory and CPUs are dedicated.

Note: For information on minimum and recommended server configurations based on the number/type of CSM Clients, see Scaling CSM.

CSM uses a dedicated message queue system to allow horizontal and vertical scaling for one or more server components. Scaling CSM enables the system to use a network of machines to distribute work, providing more resources to alleviate barriers and improve availability. For more information, see:

Assess Resource Consumption

Determine how resources such as available memory and the number of processors are being used. If the performance issue is acute, restarting a Server might provide a resolution. If the issue is chronic, resources might need to be increased or expanded.

SaaS customers must contact Cherwell Support for resource requests.

Investigate the Network Environment

If you determine that there are sufficient resources, consider the network environment, including the network path between users and the server. Pings, TraceRoutes, and Packet evaluations can help identify the existence of an issue and diagnose complications with remote locations. After investigation, if you suspect that a network issue is affecting performance, consider searching for common indicators (example: All affected users are using wireless or accessing the VPN).

Run the System Analyzer to Evaluate Business Object Structure

You can use the System Analyzer to determine which aspects of the Business Object structure are impacting performance. For example, determine if Business Object relationships are loading in a way that impacts performance. For more information, see About the System Analyzer.

Evaluate Automation Processes

Automation Processes provide great power, but if they are not optimized for best performance, they can cause heavy loads on the Automation Process Service. See Performance Considerations for Automation Processes for guidance.

Assess Memory and CPU Usage

Investigate your server memory and CPU usage to determine if either factor is affecting performance. Access this information by right-clicking the Windows Task Bar, selecting Start Task Manager, and selecting the Performance tab. When you have the data, consider the following:

Memory usage should not consistently rise above 80%. Consider that even though system memory is in the normal range, a server (example: Automation Process Server) might be running higher than expected.
CPU usage should not consistently rise above 80%. Consider that CPU does not necessarily negatively impact performance, so consider other potential issues, as well.

Common issues that impact memory and CPU include:

Setting logging to both Splunk and a file.
Running a Report that uses all Application Server resources.
Using an Automation Process continuously, which causes the Automation Process Service to use additional resources.

If you notice memory spikes when loading an SLA record, specifically, the configuration of the SLA Business Object could be causing the issue. The Breached Incidents and Open Incident alert bars in the Quick Info Tile use an aggregate expression with a Count function to determine the number of records for each category; this can cause the system to reference a large number of records, which can result in a memory leak and issues with the Application Server.

To resolve this issue, remove the aggregate expressions from the Form. If you do not want to remove the expression, consider adding an attribute with the name RecalculateSetDirty to the field; this identifies the record as saved after the expression on the field is run. You can use this solution for any fields that contain an aggregate expression with a Count Function.

Implement Database Maintenance Actions

Define database maintenance Actions using the Scheduler to perform selected database maintenance operations on a scheduled basis (for more information, see Define Database Maintenance Action Options). Using these Actions, you can:

Rebuild Full-Text Search catalog: Rebuild the Full-Text Search catalog when the database maintenance Action is run. CSM uses Full-Text Search for Quick Search and Knowledge Search. If you have problems with your index getting corrupted, you might want to rebuild the search catalog once a week or every night.
Rebuild Business Object indexes: Rebuild Business Object indexes when the database maintenance action is run. This option allows you to rebuild the indexes of the database tables that represent particular Business Objects. Then, select the Select button to select which Business Objects to re-index.
Note: By default, all Business Objects are selected. Clear Business Objects to exclude them from reindexing, or select Clear All to clear all Business Objects, and then select the ones you want to re-index. If you have a high volume of data, you might want to rebuild your table indexes monthly.
Rebuild system table indexes: Rebuild the indexes of the SQL Server tables associated with CSM system tables when the database maintenance action is run. These tables start with "Trebuchet" (the internal code name for the product). If you have a high volume of data, you might want to rebuild your table indexes monthly.
Shrink SQL event log: Shrink the SQL Server event log file when the database maintenance action is run.
Note: Consult with your Administrator before scheduling this option. If in doubt, do not use this feature unless you run into problems.
Refresh Queue status: Refresh Queue status when the database maintenance action is run. Queue status can get out of sync if Business Objects that were on Queues are deleted. This option ensures that each Queue is synchronized. We recommend scheduling this weekly.
Remove unused user accounts: Remove data associated with deleted user accounts when the database maintenance action is run. When user accounts are deleted from the system, their authorization information might not be removed. This option ensures that the authorization information is in sync with the user list by removing the unused information. We recommend scheduling this weekly.
Synchronize Team Info with team list: Synchronizes the Team Info Lookup table with CSM user and customer team list.
Run SQL Traces: Run traces on your SQL using a tool such as SQL Server Management Studio. This helps you to identify problematic or slow SQL queries that may be causing poor application performance. For more information on using SQL profilers to run SQL traces, see Microsoft documentation: https://docs.microsoft.com/en-us/sql/tools/sql-server-profiler/sql-server-profiler?view=sql-server-ver15

Tip: If you have a lookup table including between 50-100 rows with values that will not change (example: statuses such as open, closed, and so on), we recommend that you mark those Business Objects as cacheable. The system stores these records in memory and will not fetch them from the database. If you modify these values, however, you should restart the system.

Mitigate Database Growth

Use the E-mail Monitor, Blueprint Publish Log, and Automation Process Log to reduce database growth. When emails are imported into your system, large attachments can contribute to increased database size. To keep database growth caused by email attachments under control, you can prevent attachments being imported and control their size by using the E-mail Monitor Actions.

You can define the following settings for each monitor:

Attach email to [Business Object (example: Incident)]: Attaches incoming emails to Business Objects as Journal - Mail History Records.
Import attachments as part of email: Imports email attachments into the database along with incoming emails and allows you to define Attachment settings such as type and size.
Attach email attachments to [Business Object (example: Incident)]: Attaches email attachments to Business Object Record's Attachment bar (not just to the internal copy of the email).
Preserve inline images within email body: Preserves images within the body of incoming emails with the text of the email.
Attach inline images to [Business Object (example: Incident)]: Attaches images within the body of incoming emails to the selected Business Object.
Attach email to Customers: Attaches incoming emails to Customer Records as Journal - Mail History Records.

Note: For more information, see Define Monitor Item Action Options.

Using the Blueprint Publish Log (for more information, see View the Blueprint Publish Log), you can reduce the size of the TrebuchetPublishLogs system table, which holds historical publishing data. Before clearing the data, ensure that you have the information you need. Clear the log using CSM Administrator by selecting File > Clear in the Publish Log window.

Using the Automation Process Publish Log (for more information, see Monitor Automation Process Statistics), you can reduce the size of the TrebuchetProcesses system table, which holds historical publishing data. Clear the log using CSM Administrator by selecting File > Clear All Processes > Clear Completed Item History in the Automation Process Statistics window.

Investigate Browser Configurations

If the Browser Client loads more than a few seconds slower than the Desktop Client, determine if any system configurations are impacting performance. Consider the following when investigating the issue:

Server memory and CPU usage can cause performance issues with the Browser Client. For more information, review the standards in Assess Memory and CPU Usage (above).
Calendars can cause performance issues due to the amount of data that refreshes each time a Calendar is opened.
Automatic Actions can cause issues because the Save Actions are executed before the Save occurs. Prevent this from happening by creating an expression for the Actions to run if a condition is true/false and clearing the Execute Before Saving a Record check box; this allows the system to only save the record if necessary.

General Server Maintenance

Investigate your server memory usage to determine if Portal performance is being affected as a result. Access this information by right-clicking the Windows Task Bar, selecting Start Task Manager, and selecting the Processes tab. Then, you can restart the process or server using the IIS Manager.

When you restart the process or server, the memory use will temporarily decrease, but will continue to increase afterwards; this might happen because the Application Pool in IIS does not have permissions to write to the directory and instead of displaying an error, the log messages consume memory. If this happens, you have several options:

Using the Portal Logging Options window (for more information, see Configure Encryption Keys for a CSM Server or Web Application) and the Browser Logging Options window in the Cherwell Server Manager, change the file path.
Using the Application Pool Identity window in the IIS Manager, change the Identity for the Application Pool.
Enable file permissions for the Application Pool so it can write log files to your chosen location. For more information, see Securing IIS.