Cloud Run terminating long running requests when using http/2 end-to-end

[Mateusz-Słomka]

Hello,

I have an issue with Cloud Run when using the http/2 end-to-end option. When I use this option, the requests that take longer than ~60 seconds get terminated with a 503 response and no additional information. The issue is not caused by excessive resource consumption nor request rate as it always happens even for a single request at a time. I also have the Cloud Run request timeout set to a much higher number.

I use nodejs with spdy to handle http/2 requests with the following code (truncated):

const server = spdy.createServer({
  spdy: {plain: true},
  sessionTimeout: 600, 
  requestTimeout: 1000 * 3600
}, app);

server.setTimeout(1000 * 3600);
server.listen(PORT, () => {
  console.log("Listening on port: " + PORT);
});

To reliably test this case in every single request I artificially make the processing time longer by waiting more than 60 seconds before returning a response.

The issue resolves itself when I disable the http/2 end-to-end option in the cloud run, but I need it to remove the request size limit.

Is it possible that there is an issue with Cloud Run?

Thank you in advance

[Mateusz-Słomka · Accepted solution]

I've managed to resolve the issue by tweaking the 

server.headersTimeout value.

[Rohail-Zubair]

How exactly did you solve this issue? I am facing the same issue, I need a fix immediately.

[Hertz-7]

how did you exactly do this can you please elaborate?

[Hertz-7]

can you please elaborate on what you exactly did

[Mateusz-Słomka]

Hi Hertz,

If you use the spdy package in the following way:

const server = spdy.createServer(
  {
    spdy: { plain: true, protocols: ["h2"] },
  },
  app, // Express app
);

You can set the request timeout via the setTimeout method. However, it is not enough, as you also need to adjust the headersTimeout value:

server.setTimeout(1000 * 3600);
server.requestTimeout = 1000 * 3600; // This is likely unnecessary
server.headersTimeout = 1000 * 3600; // 1 hour

[Hertz-7]

The issue I'm facing is somewhat similar. I'm using HTTP streams, and my streams are getting aborted randomly when using HTTP/2. However, when I disable HTTP/2, everything works fine. I tried the suggested solution, but unfortunately, it didn’t help. 😕

I originally wanted to use HTTP/2 so I could make more than six stream calls in my application. But I’ve just realized that I don’t need a full end-to-end HTTP/2 architecture for that. Since my frontend calls are using HTTP/2 or HTTP/3 due to the server configuration, I’m already able to make more than six streams.

Still, I haven’t been able to figure out the stream aborts issue when HTTP/2 is enabled on Cloud Run.