Gemini Pro Vision API error (Unable to submit request because it has more than one contents field)

[pq111]

Hello, I am using the gemini-pro-vision LLM with image inputs, but I get the following 400 error:

Unable to submit request because it has more than one contents field but model gemini-pro-vision only supports one. Remove all but one contents and try again.. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini

This is a fake example of the contents part of my object:

contents: [{
   "role": "USER",
    "parts": [
       {"text": "what do you see in this image"},
       { "inlineData" : "(insert base64 image here)" }
    ]
}, {
    "role": "MODEL",
    "parts": [
        {"text": "be as descriptive as possible"}
    ]
}]

I cannot find any information in the API documentation about this. For the record, if I switch "MODEL" to be "USER" it works without error despite there being more than one element in the contents array. Also, if I don't use image inputs at all but still use MODEL, it also works without an error. It seems to only happen if I have a contents array with `inlineData`, more than one element. and more than one role. Is there a way around this or documentation somewhere explaining?

[cassandramae]

Hi @pq111,

Welcome to Google Cloud Community!

Would you be able to provide more details to reproduce the issue? For example, does this only happen if the last message contains an image? Just to add, note that when using inline_data parameter as a part of a multi-part Content message, you can only specify 1 image in gemini-1.0-pro-vision as mentioned here.

Thanks!