Streaming responses back with FastAPI and API Gateway

[manuathypertype]

Hi!

I have a backend service with FastAPI from which I am streaming back responses with StreamingResponse. I am serving this service from Cloud Run  and through an API Gateway back to the frontend. I have found it impossible to stream back a response because it gets buffered. Moreover, when I try to implement a SSE endpoint, it just times out. Does anyone know how to fix this?

Cheers!

[-Rhett]

Hi @manuathypertype,

Welcome to the Google Cloud Community!

Connection buffering or timeouts typically indicate network latency or configuration issues between Cloud Run and API Gateway that may be affecting the streaming. Can you share a screenshot of your API Gateway logs, metrics, or any error messages received?

To help diagnose and resolve the issue, follow these steps:

• Confirm that your service aligns with our Getting started with API Gateway and Cloud Run guide and Cloud Run general development tips
• Test your Cloud Run service locally first before deploying
• Perform General troubleshooting steps for API Gateway
• Perform General troubleshooting steps for Cloud Run
• Create and run a Connectivity test and share your test results

If you need more help, feel free to reach out through our available support channels for Google Cloud.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[manuathypertype]

Hi Rhett!

The thing is that I read online that API Gateway does not allow streaming back responses to the client. I read that it buffers the stream sent by the FastAPI StreamingResponse and then sends it altogether to the frontend. Is this correct? This is the behaviour we are seeing for a certain endpoint. So everything works for this endpoint and we do NOT see any timeouts, it's just that the response is not streamed as we expect it to be.

Then another behaviour that we see in another endpoint is that we get timeouts. This is produces because we are sending an "infinite" stream as that endpoint is a SSE endpoint.