Cloud Run Service to Service via GRPC

[dcreedon]

Hi,

I've two cloud run services,  frontend and listservice.  Both are currently open to receive traffic from all/everywhere on the internet. The call to the frontend service using HTTP then calls the listservice via GRPC. I can call both cloud run services via postman but the frontend service cannot connect/call the listservice via GRPC.  I have this working locally. And get this error on the frontend cloud run logs:  

Error: 1 CANCELLED: Call cancelled at callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:19) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:192:76) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181) at /workspace/node_modules/@grpc/grpc-js/build/src/resolving-call.js:94:78 at process.processTicksAndRejections (node:internal/process/task_queues:77:11)

I've allowed both services to be publicly accessible:

https://cloud.google.com/run/docs/authenticating/public

Any help appreciated.

[Michael_Conde]

Hi @dcreedon ,

Welcome to Google Cloud Community!

Some GCP users have observed here that calling all functions together can sometimes cause this error. A workaround they found is to execute it one at a time, it will take more time but it will resolve the error. 

If the error you are encountering is intermittent, you can also try enabling 'Retry on Failure' in Cloud function.

If error still persists, it's recommended to contact Google Support as they have appropriate tools to assist you with your query. 

Thanks

[dcreedon]

Hi, it's a single GRPC call I am testing, not multiple. The service to service GRPC has not worked yet on the Cloud Run services. It does work locally. I can connect via postman to both cloud run services and invoke the http/API on the front end and GRPC on the back end but the front end does not invoke the GRPC on the back end itself. Both services allow connections from the internet by all - no restrictions. Also I am making the GRPC call on 443 to the grpc server.

[jotaqm]

I also has this problem with nestjs. I can connect to both apps through postman or even client grpc node server locally to remote grpc cloud run. However, when deployed, cloud runs are unable to communicate with Cancelled error

[AkWwWard]

Hi. Did you manage to solve this problem? Have the same error

[AkWwWard]

Have you found a solution for this? Same here, keep retrieving "Request cancelled" when making a call via a NestJS application, but Postman calls work fine

[julien_bisconti]

Is the Cloud Run service HTTP/2 ?

I'm not sure but I think both services need HTTP/2 enabled.

Just a guess.

[jotaqm]

Hi Julien, thanks for your reply!

It started working after we configured both cloud runs to use the second generation and enabled HTTP/2. Although it may seem trivial from a network perspective, it gave us quite a headache to figure it out. Therefore, it would be nice to specify this in the documentation for better compatibility in the Cloud Run ecosystem.

Additionally, remember to keep at least one minimum instance running, as not having one could also lead to problems when a request is made, but the server is in a sleeping state.

[Nicolas-CV]

Hey, I am having the same problem with NestJS, I activated  HTTPv2 and. before I. got Call Canceled Error, and now I get: "upstream connect error or disconnect/reset before headers. reset reason: protocol error", I have the url set to the run.app url of my grpc hosting service. I have validated that calls to each individual Cloud Run is working, but when calling between them I get this reponse 

[azhtom]

I resolved the problem by enabling httpv2 and adding this config to my client 

client.options.ts:

import { credentials } from '@grpc/grpc-js'

export const grpcClientOptions: ClientOptions = {
  transport: Transport.GRPC,
  options: {
    url: `${process.env.GRPC_CLIENT_URL}`,
    package: 'auth',
    protoPath: join(__dirname, '/protos/auth.proto'),
    credentials: credentials.createSsl(),
  },
};

[silvia-seihoon]

@azhtom 

I have this solution working! Thanks 🙂

(using nesjts 10, + google cloud run http/2)