Tuning Gemini Model through VertexAI Studio - JSONL file format help

[MLskeri]

Hello, all,

I have been trying to tune a Gemini model through google cloud console using VertexAI Studio. The tuning method is 'Supervised tuning'. To tune the model, it requires a JSONL file containing the inputs and outputs for the tuning data set. I will also have a validation dataset.

I am running into an issue with the JSONL formatting which I am currently following the documentation from Google, however my data set needs to include inputting two images rather than one: https://cloud.google.com/vertex-ai/generative-ai/docs/models/tune_gemini/image_tune

When I provide my tuning dataset and validation dataset, it provides the error: Row: 0. Missing required `contents` field. I followed the documentation where it has the "contents" but the error still occurs? Does it only need ' rather than the suggested "?

The dataset format from Google it suggests for one image is:

{
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "fileData": {
            "mimeType": "image/jpeg",
            "fileUri": "gs://cloud-samples-data/ai-platform/generative_ai/gemini-1_5/image/longcap100/100.jpeg"
            }
        }, 
        {
          "text": "Describe this image in detail that captures the essence of it."
        }
      ]
    }, 
    {
      "role": "model",
      "parts": [
        {
          "text": "A man stands on a road, wearing a blue denim jacket, tan pants, and white sneakers. He has his hands in his pockets and is wearing a white t-shirt under his jacket. The man's pants are cuffed, and his shoes are white. The road is dark grey, and the leaves are green. The man is standing in the shade, and the light is shining on the ground."
        }
      ]
    }
  ]
}

Here is a section from another documentation on how to refer to multiple images:

Below is an example of 2 different examples in my tuning data set which includes two images in the user role:

{"contents": [{"role": "user", "parts": [{"fileData": {"mimeType": "image/png", "fileUri": "gs://human_rated_tactile_map_similarity_scores/NEINV002_220318_162612_DrawFullMap_01_resized.png"}}, {"text": "Image 1 (hand drawing)"}, {"fileData": {"mimeType": "image/png", "fileUri": "gs://human_rated_tactile_map_similarity_scores/Map01_solution0.png"}}, {"text": "Image 2 (template)"}, {"text": "Compare these two images and provide a similarity score from 0 to 10, where 0 is completely different and 10 means all local features are in the right locations relative to each other. Only respond with 'Similarity Score:' followed by the numeric score."}]}, {"role": "model", "parts": [{"text": "Similarity Score: 3"}]}]}
{"contents": [{"role": "user", "parts": [{"fileData": {"mimeType": "image/png", "fileUri": "gs://human_rated_tactile_map_similarity_scores/NEINV002_220318_162612_DrawFullMap_02_resized.png"}}, {"text": "Image 1 (hand drawing)"}, {"fileData": {"mimeType": "image/png", "fileUri": "gs://human_rated_tactile_map_similarity_scores/Map01_solution0.png"}}, {"text": "Image 2 (template)"}, {"text": "Compare these two images and provide a similarity score from 0 to 10, where 0 is completely different and 10 means all local features are in the right locations relative to each other. Only respond with 'Similarity Score:' followed by the numeric score."}]}, {"role": "model", "parts": [{"text": "Similarity Score: 4"}]}]}

Is anyone able to spot where the error in my dataset JSONL file? I am unsure of how to set-up the JSONL file for the validation dataset since we do not need to provide the model role for output.

Here is what I have for my validation dataset example:

{"contents": [{"role": "user", "parts": [{"fileData": {"mimeType": "image/png", "fileUri": "gs://human_rated_tactile_map_similarity_scores/NEINV001_220321_160745_DrawFullMap_01_resized.png"}}, {"text": "Image 1 (hand drawing)"}, {"fileData": {"mimeType": "image/png", "fileUri": "gs://human_rated_tactile_map_similarity_scores/Map01_solution0.png"}}, {"text": "Image 2 (template). Compare these two images and provide a similarity score from 0 to 10, where 0 is completely different and 10 means all local features are in the right locations relative to each other. Only respond with 'Similarity Score:' followed by the numeric score."}]}]}
{"contents": [{"role": "user", "parts": [{"fileData": {"mimeType": "image/png", "fileUri": "gs://human_rated_tactile_map_similarity_scores/NEINV001_220321_160745_DrawFullMap_02_resized.png"}}, {"text": "Image 1 (hand drawing)"}, {"fileData": {"mimeType": "image/png", "fileUri": "gs://human_rated_tactile_map_similarity_scores/Map01_solution0.png"}}, {"text": "Image 2 (template). Compare these two images and provide a similarity score from 0 to 10, where 0 is completely different and 10 means all local features are in the right locations relative to each other. Only respond with 'Similarity Score:' followed by the numeric score."}]}]}

Let me know if you need more information about my case. Thanks in advance!