serverStreamingPredict for finetuned models (codechat-bison)

[rmuller]

I have finetuned a codechat-bison model and it is automatically deployed in a endpoint. I can get model responses in the "predict" endpoint, but I get error 500 when using "serverStreamingPredict". Using the UI, I also cannot use streaming with the finetuned model (but I can with the base model both with the RESTAPI and using UI).

Is streaming available for finetuned models? How to configure the endpoint to enable streaming?

[lsolatorio]

Hi @rmuller, 

Thank you for reaching out to our community.

I understand that you are having challenges streaming responses using the serverStreamingPredict. The error 500 refers to an internal error which can be linked to the reported console issue and will probably resolve itself in a later time. Meanwhile, you can reference this example using the streaming REST API as it might help you review or troubleshoot your codes.

If the issue still persists, you can consider getting Vertex AI Support for better assistance.

Hope this helps.