Getting Started with Ticket Classification

Ivanti Neurons Machine Learning Lab provides you with an environment to create, train, and deploy machine learning models. This section guides you through the process of preparing and configuring Ticket Classification models for Neurons for ITSM.

What can Ticket Classification do?

When an incident is created in Neurons for ITSM, it is automatically classified real-time by a machine learning model into the correct Service, Category, and SubCategory based on the contents of the Subject field. You can train and deploy your own model by going to the Machine Learning Lab on the Neurons Platform and upload the training data.
Ticket Classification:

  • is compatible with both cloud and on-premise versions of Neurons for ITSM;
  • is available for customers who have both Neurons for ITSM and Neurons for Healing.

Ticket Classification requires your instance of Neurons for ITSM to be version 2021.3 or higher.

Configuring Ticket Classification

The process of configuring consists of the following steps:

1. Connect your instances of Neurons for ITSM and the Ivanti Neurons Platform

To connect your instances of Neurons for ITSM and the Neurons Platform, please raise a Service Request with the Ivanti Cloud Ops team (Support). They will apply the necessary configuration to Neurons for ITSM.

If you have Neurons for Healing and you have Neurons for ITSM but you do not see the item Software > Machine Learning Lab in the Neurons Platform, please contact Ivanti Support.

2. Export data to train your machine learning model

Even though Neurons Machine Learning Lab takes much of the pain out of creating machine learning models, the result also depends on the quality of the set of training data you provide. This data set will be used to train the model. Training data is nothing more than a verified set of previously created incidents with their classifications.
For the best results, the data set must cover every possible class, i.e. combination of Service, Category, and SubCategory that is available in your Neurons for ITSM environment.
If your Neurons for ITSM environment (also) uses Actual Service, Actual Category, and Actual SubCategory, and this data is more accurate, you may want to consider using these Actual-fields instead.

To export incidents for your training data set:

  1. Open the Incident workspace in Neurons for ITSM
  2. Create a query that returns the incident data you want to prepare for training your machine learning model.
  3. Go to More > Form Actions > Export.
Training data set requirements
  • The file must be in CSV format, using commas as separators.
  • The file must contain:
    • a column named Subject.
    • at least one of the columns Service, Category, or SubCategory, depending on your configuration and usage.

After exporting the data from Neurons for ITSM, you can start preparing the data set for training.
If you chose to use data from Actual Service, Actual Category, and Actual SubCategory, these columns must be renamed to Service, Category, and SubCategory respectively.

Ivanti recommends deleting all other columns from the exported CSV file.

3. Cleaning training data

Make sure you address each of the points in this step, as this can have a big impact on the accuracy of the model.
To help keep the overview, consider sorting your data on Service, Category, and SubCategory. Effectively, this creates a list sorted on class.

Cleaning the training data consists of the following actions:

  • Remove Personally Identifiable Information.
    This prevents potential data-privacy issues and (equally important for this purpose) it also prevents this sort of data from affecting the model.
    Example    : If the training data contains a subject line with a phone number, the resulting model may interpret that number as a signal for a specific class.
  • Remove or relabel deprecated classes.
    All classes in the training data must be available in Neurons for ITSM. If a model proposes a class that is not available, an error is displayed to the user and the incident may not be saved.
    If your training data contains samples of classes that have been deprecated you can:
    • relabel the sample to the correct, available class (if available).
    • delete the sample.
      Make sure your training data contains enough samples, both per class and in total.
      See For best accuracy of the trained model (below)
  • Correct or remove examples with classifications that are too generic.
    Classes containing values like "Other" or "Uncategorized" can be too generic.
    Correct these examples to more specific classes where possible and where the Subject contains information that is useful for predicting the class.
  • Remove examples where the subject contains very little or no useful information.
    Samples with e.g. the subject "Urgent request!" or "Help me ASAP" contain no information that is useful to train the model.
  • Correct any incorrect classifications.
    If an incident with the Subject "All keys fell off my keyboard" has the class "Network", correct the class to, for example, "Peripherals"
  • Remove or split examples where the Subject contains information about multiple issues in different classes.
    Users can sometimes combine completely unrelated issues into one incident, for example "My keyboard is not working and I need a VM".
    Remove this example or split it into two separate incidents, each with the correct class assigned.
  • Verify the consistency of your training data and improve it where possible.
    Incidents with near-identical (meaning of) Subject fields should have the same classification.

    Consider the following subjects:

    • "My camera is not working when I make a video call"
    • "I was in a video call when my webcam went blank"

    The classes "Videocall" and "Webcam" can both be correct, but it is important that both incidents are assigned the same class in your training data.

  • Make sure all possible classes are represented with enough examples.
    Machine Learning Lab needs enough examples for an accurate model.
    If your training data contains many classes with only a few examples, you must extend the amount of training data.
    Alternatively, consider merging and/or removing some of these classes.
  • Reduce the number of examples for classes that are represented in above average numbers.
    Such 'dominant classes' affect accuracy, because the model may become more likely to select a dominant class.
For best accuracy of the trained model

Make sure that the training data:

  • contains only known-good classifications;
  • is consistent;
  • contains (approximately) equal quantities of every class, regardless of how often each class is actually used;

  • contains a minimum of 50 samples per class. But as long as the samples fulfill the previous three criteria and the data has been properly cleaned, generally speaking more is better.

    If your environment uses 10 services and each service has 5 categories, there are 10 x 5 = 50 classes.
    The training data set should therefore be at least 50 classes x 50 examples = 2500 total examples.

    Whether the minimum amount will be enough for creating an accurate model depends on your data and your classes.

4. Prepare a stopwords file (optional)

To improve the performance of a model, you can optionally provide a stopwords file. This file contains words that the model must ignore.
A default set of stopwords from NLTK is applied automatically.

Stopwords file requirements
  • The file must be in CSV or TXT format.
  • The file must only contain one stopword per line.

5. Create and train your machine learning model

Once you have prepared your data, you can start creating and training your model.

  1. Add a new model.
  2. Upload your training data file and (optionally) stopwords file.
  3. Save your changes.
  4. Train your model.

Machine Learning Lab starts the training process.

After training has finished, the Details tab of a model will show its accuracy. Based on this accuracy, you can decide to deploy the model.
To improve accuracy, update your training data and/or stopwords.

6. Deploy your trained machine learning model

When you have a trained model, you can deploy it to your environment.
Once deployed, the model will start classifying incidents in your Neurons for ITSM environment.

7. Enable Ticket Classification in Neurons for ITSM

In Neurons for ITSM, enable Ticket Classification.
For information on how to do this, see Configuring Ticket Classification in the Neurons for ITSM configuration guide.

When to train a new model

You can choose to train a new model to increase model accuracy, for example when you have continued cleaning your set of training data.
But there are situations where training a new model can be necessary and even urgent:

  • When classes are deprecated
    If a model proposes a class that is no longer available, an error is displayed to the user in Neurons for ITSM and the incident may not be saved.
    1. Update your training data so it no longer contains examples using the deprecated class.
      Either assign a different class to the examples or remove the examples completely.
    2. Use the training data to train a new model.
    3. Deploy the new model.
    4. Deprecate the class in Neurons for ITSM.
  • When the intended use of a class changes
    Examples:
    • A class was used for landline phones before, but is now used for mobile devices.
    • A program in your environment is replaced, or a new program is introduced.
    • A new type of product (for example tablets or smartwatches) is introduced to the market, for which incidents may be created.

    Without a new model that is trained with data that also contains examples of incidents that reflect these changes, the accuracy may decline.

  • When new classes are introduced
    A model can propose only classes it is trained with. If you do not create a new model based on training data that contains the new classes, the model will never propose the new classes.