GCP Classic Application LB returns 'upgrade_header_rejected' on socket.io GET request

[matterai]

Hey!

There is a NestJS application wrapped in Docker container. It has socket.io endpoint for clients configured like this:

@WebSocketGateway({
  transport: ['websocket'],
  cors: {
    origin: '*',
    credentials: true,
  },
  allowUpgrades: true,
  pingInterval: 25000,
  pingTimeout: 20000,
})
export class SocketGateway implements OnGatewayConnection { ... }

The app deployed in GCP as Cloud Run (v2) service with pulumi. Using direct Cloud Run URL I can reach my socket endpoint and set up connection between my client (I use Postman to play around with it) and deployed service: https://<cloudrunv2service.url>/socket.io

Then I tried to configure my own Load Balancer with HTTPS certificate still usign pulumi. Everything works well except socket gateway. It returns Bad Request 400. Using Logs Explorer I figured out some details:

"jsonPayload": {
  "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
  "backendTargetProjectNumber": "projects/77577xxxxxxx"
  "remoteIp": "x.x.x.x" 
  "statusDetails": "upgrade_header_rejected"
}

I can't beat this issue. I do not understand what am I doing wrong and how I can tell to Load Balancer do not reject Upgrade header. Actually this is completely insane. 

Here is my pulumi code I used to deploy and configure load balalncer:

const backend = new gcp.compute.BackendService(`${name}-backend`, {
  name: `${name}-backend`,
  portName: 'http',
  protocol: 'HTTPS',
  sessionAffinity: 'CLIENT_IP',
  loadBalancingScheme: 'EXTERNAL',
  backends: [{ group: regionNEG.id }],
  connectionDrainingTimeoutSec: 60,
  enableCdn: false,
  customRequestHeaders: ['Connection: Upgrade', 'Upgrade: websocket'],
});

const urlMap = new gcp.compute.URLMap(`${name}-url-map`, {
    defaultService: backend.id,
    hostRules: [{ hosts: [args.domain], pathMatcher: 'allpaths' }],
    pathMatchers: [
      {
        name: 'allpaths',
        defaultService: notFoundBackend.id,
        pathRules: [
          ...
          {
            paths: ['/api', '/api/*'],
            service: backend.id,
            routeAction: {
              urlRewrite: { pathPrefixRewrite: '/' },
            },
          },
          {
            paths: ['/*'],
            service: notFoundBackend.id,
          },
        ],
      },
    ],
  }
);

const proxy = new gcp.compute.TargetHttpsProxy(`${name}-proxy`, {
  urlMap: urlMap.id,
  sslCertificates: [cert.id],
});

const address = new gcp.compute.GlobalAddress(`${name}-address`, {});

const forwardingRule = new gcp.compute.GlobalForwardingRule(
  `${name}-forwarding-rule`,
  {
    target: proxy.id,
    portRange: '443',
    ipAddress: address.address,
  },
);

Little note: I use path rewriter to remove `/api` from request path when it comes to LB. Then request comes to my Cloud Run service.

Please, give me some ideas or tell me what am I doing absolutely wrong so my websocket endpoint doesn't work. How can I configure GCP LB to stop rejecting Upgrade header? 

Thank you!

[diannemcm]

Hi @matterai,

Welcome to Google Cloud Community!

The upgrade_header_rejected that is due to rejected connection and possibly from upgrading a secure connection to an insecure one or that the header is an invalid one. 

You may look into WebSocket support, the document explains how the websocket protocol works.

Setting timeouts for both in Cloud Run and in the GCP Load Balancer to avoid premature closing of connections, websocket connection timeouts are not always the same as backend service timeouts. Websocket connection timeouts depend on the type of load balancer. 

You may also check this potential relevant thread on Stack Exchange for additional reference.

For more detailed insights you may reach out to Google Cloud Support for assistance. 

I hope the above information is helpful.