Solved: Unexpected Behavior: Gemini-1.0-Pro-002 Returns Di... - Page 2

dario_bernardo · 04-11-2024 06:51 AM

Hello,
I am encountering different LLM responses for the same prompt with the temperature set to 0.

I am using gemini-1.0-pro-002, and I have noticed that, for some reason, setting the temperature to 0 does not always result in the LLM returning the same response. I have verified this through both the Python API and the GCP Vertex AI web interface. This issue does not seem to affect the 001 version.

I believe this to be a bug. Thank you

lsolatorio

Hi @dario_bernardo,

We appreciate you taking the time to share your observations about Gemini-1.0-Pro-002. It helps us improve!

Every model release builds on the previous one. Gemini-1.0-Pro-002 was trained with a much larger dataset, allowing for more variations in its responses. This, along with the slight non-determinism even at temperature 0 (as mentioned in the 'Send multimodal prompt requests' document), can explain the differences you're seeing.

A temperature of 0 means that the highest probability tokens are always selected. In this case, responses for a given prompt are mostly deterministic, but a small amount of variation is still possible.

If you feel this behavior is a bug, you can submit a ticket in our issue tracking system.

I hope I was able to provide you with useful insights.

View solution in original post

Unexpected Behavior: Gemini-1.0-Pro-002 Returns Different Outputs at Temperature 0