I want to monitor user input text with Vertex AI Model Monitoring.

[YoshinaoMori]

I am using the ML model to predict toxicity or not for a user entered text. If I enable Vertex AI Model Monitoring for that user-entered text, can I successfully detect skew and drift?
I am thinking that the string type data is treated as a categorical feature and the user entered text is not the same as the training data, which would trigger an alert for skew or drift.

If not, is there a way to detect skew and drift for different user-entered text each time?

[Roderick]

Hi @YoshinaoMori  - Yes, you can use Vertex AI Model Monitoring to detect skew and drift for different user-entered text each time. Vertex AI Model Monitoring uses a variety of metrics to track the performance of your model, including:

• Accuracy: The accuracy of your model is the percentage of times that it correctly predicts the toxicity of a text.
• Precision: The precision of your model is the percentage of times that it predicts toxicity when the text is actually toxic.
• Recall: The recall of your model is the percentage of times that it predicts toxicity when the text is actually toxic.
• Skew: Skew is the difference between the distribution of toxic and non-toxic text in your training data and the distribution of toxic and non-toxic text in your test data.
• Drift: Drift is the change in the distribution of toxic and non-toxic text over time.

If the accuracy, precision, or recall of your model decreases, or if the skew or drift of your model increases, Vertex AI Model Monitoring will trigger an alert. This will allow you to take action to address the issue and keep your model performing at its best.

Here are some additional things you can do to detect skew and drift for different user-entered text each time:

• Use a large training dataset: The larger your training dataset, the more likely it is that your model will be able to generalize to new data.
• Use a diverse training dataset: The more diverse your training dataset, the more likely it is that your model will be able to handle different types of text.
• Monitor your model's performance over time: Vertex AI Model Monitoring can help you track the performance of your model over time. This will help you to identify any changes in the distribution of toxic and non-toxic text, and take action to address the issue.

I hope this helps!

[YoshinaoMori]

@Roderick 

Sorry for the late reply.

I used model monitoring.

I ask this question because I have been having trouble monitoring the model.

We have 150 text data as training data; each of the 150 text data is different. We created a model using this training data, deployed it to an endpoint, and enabled model monitoring.

We selected one piece of data from the training data and changed one of the letters from lowercase to uppercase. Since this changed text only differs by one letter from the training data, we expect that it will not be detected as skew. We inferred this changed text on the endpoints. This changed text was then detected as skew. The threshold is 0.3 and the skew value is detected as 0.4.

Is this method correct?

Also, when the same text as the training data was inferred on the endpoints, it was also detected as skew. Does this affect the number of times the text appears in the training data?

Also, I understand that you can monitor with Accuracy, Precision, and Recall, how do I set this up?