LinkedIn
Twitter
Hello,
We are proxying the Azure Open AI endpoints via ApigeeX. When streaming is enabled on the Open AI request, Azure sends the response as SSE with content-type being text/event-stream. Even though the message structure and content-type is getting passed on correctly from ApigeeX to client, the Wait Time(TTFB) is taking too long indicating Apigee is not sending the first byte as soon as it receives from Azure endpoint, but rather buffering it and responding with all chunks at once in the end. I have also enabled both response and request streaming on proxy and target endpoints, but it did not fix the issue.
Has anyone proxied a SSE target with Apigee?
I have attached the screenshots showing the difference in behaviour.
Azure Wait time
Apigee Wait Time
