Error while deploying hugging pytorch model (ROBERTA) to Vertex AI

[kjagdish696]

Hello

I am new to the vertex AI. I have created a binary classfication model using hugging face Pytorch model (ROBERTA).  Now I am following the google vertex AI documentation to deploy but I am facing somer errors while deploying.

ERROR 2023-05-22T08:08:18.241378784Z usage: torchserve [-h] [-v | --start | --stop] [--ts-config TS_CONFIG]
ERROR 2023-05-22T08:08:18.241437196Z [--model-store MODEL_STORE]
ERROR 2023-05-22T08:08:18.241444587Z [--workflow-store WORKFLOW_STORE]
ERROR 2023-05-22T08:08:18.241451025Z [--models MODEL_PATH1 MODEL_NAME=MODEL_PATH2... [MODEL_PATH1 MODEL_NAME=MODEL_PATH2... ...]]
ERROR 2023-05-22T08:08:18.241456985Z [--log-config LOG_CONFIG] [--foreground]
ERROR 2023-05-22T08:08:18.241462945Z [--no-config-snapshots] [--plugins-path PLUGINS_PATH]
ERROR 2023-05-22T08:08:18.241468906Z torchserve: error: unrecognized arguments: --handler /home/jupyter/Container/hugging_face_handler.py
ERROR 2023-05-22T08:08:22.568594694Z usage: torchserve [-h] [-v | --start | --stop] [--ts-config TS_CONFIG]
ERROR 2023-05-22T08:08:22.568649768Z [--model-store MODEL_STORE]
ERROR 2023-05-22T08:08:22.568657159Z [--workflow-store WORKFLOW_STORE]
ERROR 2023-05-22T08:08:22.568664073Z [--models MODEL_PATH1 MODEL_NAME=MODEL_PATH2... [MODEL_PATH1 MODEL_NAME=MODEL_PATH2... ...]]
ERROR 2023-05-22T08:08:22.568670034Z [--log-config LOG_CONFIG] [--foreground]
ERROR 2023-05-22T08:08:22.568676233Z [--no-config-snapshots] [--plugins-path PLUGINS_PATH]
ERROR 2023-05-22T08:08:22.568681955Z torchserve: error: unrecognized arguments: --handler /home/jupyter/Container/hugging_face_handler.py

Docker Image code

FROM pytorch/torchserve:latest

# Install additional dependencies if required
# RUN pip install transformers

# Copy the requirements file to the container
COPY requirements.txt /home/jupyter/Container/requirements.txt

# Install the requirements
RUN pip install -r /home/jupyter/Container/requirements.txt
# Copy your model and inference code to the container
COPY cls /home/jupyter/Container/cls
COPY hugging_face_handler.py /home/jupyter/Container/hugging_face_handler.py

# Set the working directory
WORKDIR /home/jupyter/Container/

# Expose the port used by TorchServe (default: 8080)
EXPOSE 8080

# Start TorchServe with your custom model and handler
CMD ["torchserve", "--start", "--model-store", ".", "--models", "my_model=/home/jupyter/Container/cls/pytorch_model.bin", "--handler", "/home/jupyter/Container/hugging_face_handler.py"]

Handler file code:

from transformers import RobertaForSequenceClassification, RobertaTokenizer
import torch
from sklearn import preprocessing
import numpy as np

class TransformersClassifierHandler(BaseHandler):
"""
The handler takes an input string and returns the classification text
based on the serialized transformers checkpoint.
"""
def __init__(self):
super(TransformersClassifierHandler, self).__init__()
self.initialized = False
self.model = None
self.tokenizer = None
self.device = None

def initialize(self, ctx):
# self.manifest = ctx.manifest
# properties = ctx.system_properties
model_dir = "/home/jupyter/Container/cls" # Set the model directory path
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load model
self.model = RobertaForSequenceClassification.from_pretrained(model_dir)
self.model.to(self.device)
self.model.eval()
logger.debug('Transformer model from path {0} loaded successfully'.format(model_dir))

# Ensure to use the same tokenizer used during training
self.tokenizer = RobertaTokenizer.from_pretrained(model_dir)

self.initialized = True

def preprocess(self, data):
text = data.get("data", [""])[0]
sentences = text.decode('utf-8')
logger.info("Received text: '%s'", sentences)

# Tokenize the texts
tokenizer_args = ((sentences,))
inputs = self.tokenizer(*tokenizer_args,
padding='max_length',
max_length=512,
truncation=True,
return_tensors="pt")
return inputs

def inference(self, inputs):
with torch.no_grad():
outputs = self.model(**inputs)
logits = outputs.logits
probabilities = torch.softmax(logits, dim=1)
predicted_labels = torch.argmax(probabilities, dim=1).item()
return ['Y' if predicted_labels == 1 else 'N']

def postprocess(self, inference_output):
return inference_output

I am not sure where I am going wrong over here and getting the above mentioned error. So any help in this topic would be of great help

Thanks

Jagdish

(PII Removed by Staff)

(PII Removed by Staff)

[kvandres]

Good day @kjagdish696,

Welcome to Google Cloud Community!

This error is due to the --handler argument when you start the torchserve with the custom model. Please note that you need the model binary when running the torchserve. In your code, Before running the torchserve, you need  to create a model archive file and before exposing the ports, you need to create a torchserve configuration file. You can see the sample codes below:

how to create the config file:

USER root
RUN printf "\nservice_envelope=json" >> /home/model-server/config.properties
RUN printf "\ninference_address=http://0.0.0.0:8080" >> /home/model-server/config.properties
USER model-server

how to create a model archive file: 

RUN torch-model-archiver -f \
  --model-name=<application name> \ 
  --version=1.0 \ 
  --serialized-file=<path of.bin file> \
  --handler=<path of handler>
  --extra-files=<additional files referenced by the handler>

After that you can run the torchserve and you don't need to specify the --handler argument. The .mar file is the archive file format that was created earlier.

CMD ["torchserve", \
     "--start", \
     "--ts-config=/home/model-server/config.properties", \
     "--models", \
     "<app_name>=<app_name>.mar", \
     "--model-store", \
     "/home/model-server/model-store"]

You may have also missed some steps, here is the step by step process on how to deploy Pytorch Model to Vertex AI: https://cloud.google.com/blog/topics/developers-practitioners/pytorch-google-cloud-how-deploy-pytorc...
https://cloud.google.com/blog/topics/developers-practitioners/pytorch-google-cloud-how-train-and-tun...

Hope this helps!