How to use Gemma in vertex AI?

[Ruoshan]

I am using Gemma in vertex AI. But I meet the following problems:

1) About the format of instance in python, according to the official documentation(https://cloud.google.com/vertex-ai/generative-ai/docs/text/test-text-prompts#generative-ai-test-text...), it should look like the following, however, it will return:"google.api_core.exceptions.FailedPrecondition: 400 Failed to deserialize the JSON body into the target type: instances[0]: missing field `inputs` at line 1 column 41"

instances =  {
        "prompts": "What is a car?",
    }

parameters_dict = {
        "temperature": 0.0, 
        "max_output_tokens":5,
        "top_p": 1,
        "top_k": 1, 
}

Therefore, I use "inputs" instead of "prompt". Is this normal?

2) my second question is the returned results keep changing even if I set the temperature as 0.0. Additionally, the max_out_tokens doesn't work, it always outputs a lot which obviously exceed the number I set.

3) Is it possible to set a stop_string as a hyperparameter to use this gemma model, e.g.

parameters_dict = {
        "temperature": 0.0,  
        "max_output_tokens":5, 
        "top_p": 1,  
        "top_k": 1,  
        "stop_string": "Result:"
}

 4) I realized the gemma model is very slow, it even takes  several nearly 1 minute for a simple question(I test ). It there any way to improve it?

Can anyone help me? Thanks !

[lsolatorio]

Hi @Ruoshan, 

Welcome and appreciate you for reaching out to our community.

I understand that you are having challenges using Gemma in Vertex AI, one thing I noticed is the sample you have provided falls under the REST test text prompts, you may need to look into the specific code guides written in python for the proper format. 

As for your other questions, I am not so sure how to address them since I haven't tried them myself however let me share with you some resources that might be of help. 

• Gemma with Vertex AI
• Gemma formatting and system instructions
• Fine-tune Gemma using KerasNLP and deploy to Vertex AI

I hope this will be of assistance to you.

[timbosssds]

Hi,

May I check did you ‘deploy’ Gemma in vertex ai? If so, can you

1. Explain how (one click deploy is giving me errors)

2. Where you able to then use gemma from a vertex ai notebook (if so, can you share code)?