How to understand the JSON returned from Gemini with respect to citations

[lambdalove]

I'd like some help understanding the meaning of the JSON returned by Gemini with respect to citations. (I checked Google's API documentation, but I found it to be sparse and unclear.)

I've grounded Gemini 1.5 Pro in a Vertex AI Search app and data store. I've observed that when I prompt the model, in the JSON response may contain the following info pertaining to citations:

1) candidates[].groundingMetadata.groundingChunks[]
This is a list of objects with a retrievedContext property, which indicates the title and URI of a particular document in the data store.

2) candidates[].groundingMetadata.groundingSupports[]
As far as I've seen, this list contains a single object, which includes:
-- the model's text response to the prompt (segment.text)
-- a groundingChunkIndices[] list (Google's documentation indicates that this is a list of indexes into the groundingChunks[] list mentioned above.)

My questions:
Can someone please explain the actual meaning of groundingChunks[] vs. groundingChunkIndices[]? I don't know how to interpret these fields in terms of what they mean for citations.

For example, I've seen something like the following:

"groundingChunks": [
  {
    retrievedContext: {
      uri: 'gs://my-bucket/0.pdf',
      title: 'A'
    }
  },
  {
    retrievedContext: {
      uri: 'gs://my-bucket/1.pdf',
      title: 'B'
    }
  },
  {
    retrievedContext: {
      uri: 'gs://my-bucket/2.pdf',
      title: 'C'
    }
  },
  {
    retrievedContext: {
      uri: 'gs://my-bucket/3.pdf',
      title: 'D'
    }
  },
  {
    retrievedContext: {
      uri: 'gs://my-bucket/4.pdf',
      title: 'E'
    }
  }
]

"groundingSupports": [
  {
    segment: {
      endIndex: 69,
      text: '(Here is the text that the model produced in response to the prompt.)'
    },
    groundingChunkIndices: [ 0, 3 ],
    confidenceScores: [ 0.9919262, 0.9919262 ]
  }
]

As you can see, there are five documents in groundingChunks[], two of which are specified in groundingChunkIndices[]. What does this mean?
Did the model produce a response based on all 5 documents in groundingChunks[]? But then what does it mean that just two of them are indicated in groundingChunkIndices[]?

Thanks for your help!

[gimaldi]

Ciao @lambdalove,

it seems to me that in the second part of your output, the one relating to groundingSupports, the startIndex key is missing. A complete response must be something like:

"groundingChunks": [
  {
    "retrievedContext": {
      "uri": "gs://my-data-bucket/dmv.txt",
      "title": "dmv"
    }
  }
],
"groundingSupport": [
  {        
    "segment": {        
      "startIndex": 294,        
      "endIndex": 439        
    },        
    "segment_text": "ipsum lorem ...",        
    "supportChunkIndices": [0],        
    "confidenceScore": [0.9541752, 0.9325467]        
  }
]

Anyway, in your case the model produced an answer based on all 5 documents. From chunks 0 and 3, i.e. 0.pdf and 3.pdf, he copied some text and inserted it into the response verbatim. In fact it is a citation from the original text. The startIndex and endIndex keys indicate which part of the response is reproduced verbatim from the grounding texts. In the JSON above, "segment_text" text content, starting from character 294 till character 439, is copied from the grounding document.

A simple example:

"segment": {        
  "startIndex": 13,
  "endIndex": 28
}
"segment_text": "Hello world, I'm happy to be here."

The resulting citation is I'm happy to be.

Hope it helps.

Ciao

[McMaco]

Hello lambdalove,

I agree with the statement of gimaldi, he copied some text in those indices which are [0, 3 ] and inserted it into the response.

Based on the documentation with regards to GenerateContentResponse specifically in the GroundingSupport field of groundingChunkIndices[] is a list of indices (into 'grounding_chunk') specifying the citations associated with the claim. For instance [1,3,4] means that grounding_chunk[1], grounding_chunk[3], grounding_chunk[4] are the retrieved content attributed to the claim.

Again, the model did not produce a response based on all 5 documents in groundingChunks[], but only in 2 documents which are [0,3] to be specific.

I hope the above information is helpful.

[lambdalove]

Thank you @gimaldi  and @McMaco for your replies, I really appreciate this help.

OK, so I now understand that "groundingSupports" identifies where the model has used, in its response, verbatim text from one or more of the grounding documents.

Can you please clarify what is meant by the documents listed "groundingChunks"? I'm still not entirely clear on this.

For example, let's consider a document that appears in "groundingChunks", but is NOT referenced in "groundingSupports". In this case, we know the model didn't verbatim quote the document in its response... Did the model find information in the document that informed or supported its response?

Thanks!