Cloud Build timeout

[Yopin]

Hi!

We are trying to deploy our App to App Engine using Cloud build, it was working without problems taking around ~5min each compilation, but now it's getting stuck on the npm install step taking 1hr to finally fail by timeout.

Thanks

[Lawrence3]

I'm also facing the same issue. All our builds are failing.

[Roopa_M]

Can you share the error that you are getting from the cloud console and minimal steps to replicate.

[Lawrence3]

Hi Roopa, the build used to take 6 mins to complete but now timeouts after 12 minutes (even after increasing timeout in cloudbuild.yaml to 900s). I tried both npm and yarn nevertheless the same issue. Below are the logs

NPM logs

Step #0: Step 1/19 : FROM node:22
Step #0: 22: Pulling from library/node
Step #0: ca4e5d672725: Pulling fs layer
Step #0: 30b93c12a9c9: Pulling fs layer
Step #0: 10d643a5fa82: Pulling fs layer
Step #0: d6dc1019d793: Pulling fs layer
Step #0: 63824f9ef397: Pulling fs layer
Step #0: 50fc95442b3d: Pulling fs layer
Step #0: 9e3b63b7d038: Pulling fs layer
Step #0: 18bb0bf9b4dc: Pulling fs layer
Step #0: d6dc1019d793: Waiting
Step #0: 63824f9ef397: Waiting
Step #0: 50fc95442b3d: Waiting
Step #0: 9e3b63b7d038: Waiting
Step #0: 18bb0bf9b4dc: Waiting
Step #0: 30b93c12a9c9: Verifying Checksum
Step #0: 30b93c12a9c9: Download complete
Step #0: ca4e5d672725: Verifying Checksum
Step #0: ca4e5d672725: Download complete
Step #0: 63824f9ef397: Verifying Checksum
Step #0: 63824f9ef397: Download complete
Step #0: 10d643a5fa82: Verifying Checksum
Step #0: 10d643a5fa82: Download complete
Step #0: 9e3b63b7d038: Verifying Checksum
Step #0: 9e3b63b7d038: Download complete
Step #0: 18bb0bf9b4dc: Verifying Checksum
Step #0: 18bb0bf9b4dc: Download complete
Step #0: 50fc95442b3d: Verifying Checksum
Step #0: 50fc95442b3d: Download complete
Step #0: d6dc1019d793: Verifying Checksum
Step #0: d6dc1019d793: Download complete
Step #0: ca4e5d672725: Pull complete
Step #0: 30b93c12a9c9: Pull complete
Step #0: 10d643a5fa82: Pull complete
Step #0: d6dc1019d793: Pull complete
Step #0: 63824f9ef397: Pull complete
Step #0: 50fc95442b3d: Pull complete
Step #0: 9e3b63b7d038: Pull complete
Step #0: 18bb0bf9b4dc: Pull complete
Step #0: Digest: sha256:86915971d2ce1548842315fcce7cda0da59319a4dab6b9fc0827e762ef04683a
Step #0: Status: Downloaded newer image for node:22
Step #0:  ---> b966edc80bd3
Step #0: Step 2/19 : RUN mkdir app
Step #0:  ---> Running in 1030f27de4c4
Step #0: Removing intermediate container 1030f27de4c4
Step #0:  ---> c4ea43ffec1c
Step #0: Step 3/19 : ADD . /app
Step #0:  ---> 63cb07e12a8a
Step #0: Step 4/19 : WORKDIR /app
Step #0:  ---> Running in c84be1d98c39
Step #0: Removing intermediate container c84be1d98c39
Step #0:  ---> 84953f5a7d64
Step #0: Step 5/19 : RUN npm clean-install --audit false --progress=false
Step #0:  ---> Running in a775838e7478
Step #0: npm warn deprecated w3c-hr-time@1.0.2: Use your platform's native performance.now() and performance.timeOrigin.
Step #0: npm warn deprecated npmlog@5.0.1: This package is no longer supported.
Step #0: npm warn deprecated subscriptions-transport-ws@0.11.0: The `subscriptions-transport-ws` package is no longer maintained. We recommend you use `graphql-ws` instead. For help migrating Apollo software to `graphql-ws`, see https://www.apollographql.com/docs/apollo-server/data/subscriptions/#switching-from-subscriptions-transport-ws    For general help using `graphql-ws`, see https://github.com/enisdenjo/graphql-ws/blob/master/README.md
Step #0: npm warn deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
Step #0: npm warn deprecated google-p12-pem@3.1.4: Package is no longer maintained
Step #0: npm warn deprecated gauge@3.0.2: This package is no longer supported.
Step #0: npm warn deprecated domexception@4.0.0: Use your platform's native DOMException instead
Step #0: npm warn deprecated fstream@1.0.12: This package is no longer supported.
Step #0: npm warn deprecated puppeteer@21.11.0: < 22.8.2 is no longer supported
Step #0: npm warn deprecated are-we-there-yet@2.0.0: This package is no longer supported.
Step #0: npm warn deprecated abab@2.0.6: Use your platform's native atob() and btoa() methods instead
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated rimraf@2.7.1: Rimraf versions prior to v4 are no longer supported
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: npm warn deprecated superagent@8.1.2: Please upgrade to v9.0.0+ as we have fixed a public vulnerability with formidable dependency. Note that v9.0.0+ requires Node.js v14.18.0+. See https://github.com/ladjs/superagent/pull/1800 for insight. This project is supported and maintained by the team at Forward Email @ https://forwardemail.net
Step #0: 
Step #0: added 1812 packages in 11m
TIMEOUT
ERROR: context deadline exceeded

 Yarn logs:

Step #0: Step 1/19 : FROM node:22
Step #0: 22: Pulling from library/node
Step #0: ca4e5d672725: Pulling fs layer
Step #0: 30b93c12a9c9: Pulling fs layer
Step #0: 10d643a5fa82: Pulling fs layer
Step #0: d6dc1019d793: Pulling fs layer
Step #0: 63824f9ef397: Pulling fs layer
Step #0: 50fc95442b3d: Pulling fs layer
Step #0: 9e3b63b7d038: Pulling fs layer
Step #0: 18bb0bf9b4dc: Pulling fs layer
Step #0: d6dc1019d793: Waiting
Step #0: 63824f9ef397: Waiting
Step #0: 50fc95442b3d: Waiting
Step #0: 9e3b63b7d038: Waiting
Step #0: 18bb0bf9b4dc: Waiting
Step #0: 30b93c12a9c9: Verifying Checksum
Step #0: 30b93c12a9c9: Download complete
Step #0: ca4e5d672725: Verifying Checksum
Step #0: ca4e5d672725: Download complete
Step #0: 10d643a5fa82: Verifying Checksum
Step #0: 10d643a5fa82: Download complete
Step #0: 63824f9ef397: Verifying Checksum
Step #0: 63824f9ef397: Download complete
Step #0: 9e3b63b7d038: Download complete
Step #0: 18bb0bf9b4dc: Verifying Checksum
Step #0: 18bb0bf9b4dc: Download complete
Step #0: 50fc95442b3d: Verifying Checksum
Step #0: 50fc95442b3d: Download complete
Step #0: d6dc1019d793: Verifying Checksum
Step #0: d6dc1019d793: Download complete
Step #0: ca4e5d672725: Pull complete
Step #0: 30b93c12a9c9: Pull complete
Step #0: 10d643a5fa82: Pull complete
Step #0: d6dc1019d793: Pull complete
Step #0: 63824f9ef397: Pull complete
Step #0: 50fc95442b3d: Pull complete
Step #0: 9e3b63b7d038: Pull complete
Step #0: 18bb0bf9b4dc: Pull complete
Step #0: Digest: sha256:86915971d2ce1548842315fcce7cda0da59319a4dab6b9fc0827e762ef04683a
Step #0: Status: Downloaded newer image for node:22
Step #0:  ---> b966edc80bd3
Step #0: Step 2/19 : RUN mkdir app
Step #0:  ---> Running in ea9e18cf2f2a
Step #0: Removing intermediate container ea9e18cf2f2a
Step #0:  ---> 2fc7611852ad
Step #0: Step 3/19 : ADD . /app
Step #0:  ---> 2af312b0320a
Step #0: Step 4/19 : WORKDIR /app
Step #0:  ---> Running in d83154e8469d
Step #0: Removing intermediate container d83154e8469d
Step #0:  ---> 3c5a5b49a8b3
Step #0: Step 5/19 : RUN yarn install
Step #0:  ---> Running in 7b34b8b192ae
Step #0: yarn install v1.22.22
Step #0: info No lockfile found.
Step #0: warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
Step #0: [1/4] Resolving packages...
Step #0: warning @ntegral/nestjs-sentry > @nestjs/graphql > subscriptions-transport-ws@0.11.0: The `subscriptions-transport-ws` package is no longer maintained. We recommend you use `graphql-ws` instead. For help migrating Apollo software to `graphql-ws`, see https://www.apollographql.com/docs/apollo-server/data/subscriptions/#switching-from-subscriptions-transport-ws    For general help using `graphql-ws`, see https://github.com/enisdenjo/graphql-ws/blob/master/README.md
Step #0: warning bcrypt > @mapbox/node-pre-gyp > npmlog@5.0.1: This package is no longer supported.
Step #0: warning bcrypt > @mapbox/node-pre-gyp > rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
Step #0: warning bcrypt > @mapbox/node-pre-gyp > rimraf > glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: warning bcrypt > @mapbox/node-pre-gyp > npmlog > are-we-there-yet@2.0.0: This package is no longer supported.
Step #0: warning bcrypt > @mapbox/node-pre-gyp > npmlog > gauge@3.0.2: This package is no longer supported.
Step #0: warning bcrypt > @mapbox/node-pre-gyp > rimraf > glob > inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
Step #0: warning exceljs > archiver > archiver-utils > glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: warning exceljs > archiver > zip-stream > archiver-utils > glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: warning exceljs > unzipper > fstream@1.0.12: This package is no longer supported.
Step #0: warning exceljs > unzipper > fstream > rimraf@2.7.1: Rimraf versions prior to v4 are no longer supported
Step #0: warning exceljs > unzipper > fstream > rimraf > glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: warning puppeteer@21.11.0: < 22.8.2 is no longer supported
Step #0: warning @Google/clasp > google-auth-library > gtoken > google-p12-pem@3.1.4: Package is no longer maintained
Step #0: warning @japa/api-client > superagent@8.1.2: Please upgrade to v9.0.0+ as we have fixed a public vulnerability with formidable dependency. Note that v9.0.0+ requires Node.js v14.18.0+. See https://github.com/ladjs/superagent/pull/1800 for insight. This project is supported and maintained by the team at Forward Email @ https://forwardemail.net
Step #0: warning @japa/assert > api-contract-validator > api-schema-builder > swagger-parser > @apidevtools/swagger-parser > z-schema@5.0.6: has issues with node 14
Step #0: warning @nestjs/cli > fork-ts-checker-webpack-plugin > memfs@3.6.0: this will be v4
Step #0: warning jest > @jest/core > jest-runtime > glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: warning jest > jest-cli > jest-config > glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: warning jest > @jest/core > @jest/reporters > glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: warning jest > @jest/core > @jest/transform > babel-plugin-istanbul > test-exclude > glob@7.2.3: Glob versions prior to v9 are no longer supported
Step #0: warning maildev > rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
Step #0: warning maildev > jsdom > abab@2.0.6: Use your platform's native atob() and btoa() methods instead
Step #0: warning maildev > jsdom > data-urls > abab@2.0.6: Use your platform's native atob() and btoa() methods instead
Step #0: warning maildev > jsdom > domexception@4.0.0: Use your platform's native DOMException instead
Step #0: warning maildev > jsdom > w3c-hr-time@1.0.2: Use your platform's native performance.now() and performance.timeOrigin.
Step #0: [2/4] Fetching packages...
Step #0: [3/4] Linking dependencies...
Step #0: warning " > @nestjs-modules/mailer@2.0.2" has unmet peer dependency "@types/ejs@>=3.0.3".
Step #0: warning " > @nestjs-modules/mailer@2.0.2" has unmet peer dependency "@types/pug@>=2.0.6".
Step #0: warning " > @nestjs-modules/mailer@2.0.2" has unmet peer dependency "@types/mjml@>=4.7.4".
Step #0: warning " > @nestjs-modules/mailer@2.0.2" has unmet peer dependency "ejs@>=3.1.2".
Step #0: warning " > @nestjs-modules/mailer@2.0.2" has unmet peer dependency "mjml@>=4.15.3".
Step #0: warning " > @nestjs-modules/mailer@2.0.2" has unmet peer dependency "pug@>=3.0.1".
Step #0: warning " > @nestjs-modules/mailer@2.0.2" has unmet peer dependency "liquidjs@>=10.8.2".
Step #0: warning " > @nestjs-modules/mailer@2.0.2" has unmet peer dependency "preview-email@>=3.0.19".
Step #0: warning " > @ntegral/nestjs-sentry@4.0.1" has unmet peer dependency "@sentry/hub@^7.7.0".
Step #0: warning " > @ntegral/nestjs-sentry@4.0.1" has incorrect peer dependency "reflect-metadata@^0.1.13".
Step #0: warning " > @ntegral/nestjs-sentry@4.0.1" has incorrect peer dependency "rimraf@^3.0.2".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql@12.0.11" has unmet peer dependency "graphql@^16.6.0".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql@12.0.11" has incorrect peer dependency "reflect-metadata@^0.1.13".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > @nestjs/mapped-types@2.0.2" has incorrect peer dependency "reflect-metadata@^0.1.12".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > graphql-tag@2.12.6" has unmet peer dependency "graphql@^0.9.0 || ^0.10.0 || ^0.11.0 || ^0.12.0 || ^0.13.0 || ^14.0.0 || ^15.0.0 || ^16.0.0".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > graphql-ws@5.14.2" has unmet peer dependency "graphql@>=0.11 <=16".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > subscriptions-transport-ws@0.11.0" has unmet peer dependency "graphql@^15.7.2 || ^16.0.0".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > @graphql-tools/utils@10.0.8" has unmet peer dependency "graphql@^14.0.0 || ^15.0.0 || ^16.0.0 || ^17.0.0".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > @graphql-tools/merge@9.0.0" has unmet peer dependency "graphql@^14.0.0 || ^15.0.0 || ^16.0.0 || ^17.0.0".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > @graphql-tools/merge > @graphql-tools/utils@10.3.2" has unmet peer dependency "graphql@^14.0.0 || ^15.0.0 || ^16.0.0 || ^17.0.0".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > @graphql-tools/schema@10.0.0" has unmet peer dependency "graphql@^14.0.0 || ^15.0.0 || ^16.0.0 || ^17.0.0".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > @graphql-tools/schema > @graphql-tools/merge@9.0.4" has unmet peer dependency "graphql@^14.0.0 || ^15.0.0 || ^16.0.0 || ^17.0.0".
Step #0: warning "@ntegral/nestjs-sentry > @nestjs/graphql > @graphql-tools/utils > @graphql-typed-document-node/core@3.2.0" has unmet peer dependency "graphql@^0.8.0 || ^0.9.0 || ^0.10.0 || ^0.11.0 || ^0.12.0 || ^0.13.0 || ^14.0.0 || ^15.0.0 || ^16.0.0 || ^17.0.0".
Step #0: warning " > @algoan/nestjs-logging-interceptor@2.6.0" has unmet peer dependency "express@>=4".
Step #0: warning " > @apidevtools/swagger-parser@10.1.0" has unmet peer dependency "openapi-types@>=7".
Step #0: warning "@japa/assert > api-contract-validator > api-schema-builder > swagger-parser > @apidevtools/swagger-parser@10.0.3" has unmet peer dependency "openapi-types@>=7".
Step #0: warning " > @japa/run-failed-tests@1.1.1" has incorrect peer dependency "@japa/runner@^2.2.3".
Step #0: warning " > ts-loader@9.5.1" has unmet peer dependency "webpack@^5.0.0".
Step #0: warning Workspaces can only be enabled in private projects.
Step #0: [4/4] Building fresh packages...

[Yopin]

The build was getting stucked in the npm install step. I added a --verbose flag and found out there was a problem with puppeteer dependency post-install script.

I ended up changing the puppeteer dependency for puppeteer-core.

This is the log of a failed build execution

Step #2: npm info run @nestjs/core@10.3.0 postinstall node_modules/@nestjs/core opencollective || exit 0
Step #2: npm info run protobufjs@7.3.2 postinstall node_modules/protobufjs node scripts/postinstall
Step #2: npm info run puppeteer@21.7.0 postinstall node_modules/puppeteer node install.mjs
Step #2: npm info run protobufjs@7.3.2 postinstall { code: 0, signal: null }
Step #2: npm info run @nestjs/core@10.3.0 postinstall { code: 0, signal: null }

[Lawrence3]

Upgrading puppeteer to the latest version (v23+) worked for me.

[rsaffer]

This fixed for me too!

I am using firebase functions gen 2. I tried to add this --verbose flag myself but cannot work out how to do it, since it seems the firebase CLI handles all this for me including running the `npm ci --quiet` command. Any tips so I can fix issues like this myself in the future?

[Evopass]

I'm with the same problem, the 3 days ago started this problem. Did you find a solution?

` 

`

[Lawrence3]

This might be late but the temporal workaround is to do npm install in your dev environment, compress the "node_modules" folder and upload to cloud storage (or any storage service). In the Dockerfile, instead of npm/yarn install, download node_modules folder with curl.

In dev environment:

# compress the node_modules dir
tar -zcvf node_modules.tar.gz node_modules/

# upload to gcloud storage (can use s3), with public access for easy download in the Dockerfile
gcloud storage cp node_modules.tar.gz gs://<bucket>/destination --canned-acl=publicRead

Dockerfile

# RUN npm install - replaced with curl
RUN curl -O https://storage.googleapis.com/<bucket><destination-path>/node_modules.tar.gz

# Extract the contents of the tar.gz file, this creates node_modules dir
RUN tar -xzf node_modules.tar.gz

...
...

I hope this helps

[SamiPussinen]

We also started getting this with Google Cloud Functions, which uses Cloud Build. The exact same step hangs for us: "Building fresh packages..."

We have tried downgrading from node 20 to node 18, tried with Cloud Functions Gen1 and Gen2 and also fiddling with dependencies. Nothing seems to resolve the issue. This just started happening out of the blue without any changes on our side.

Did anyone figure this one out? Lawrences answer could be a workaround, but we'd like the thing to work as intended instead of complicating the build process further.

[Roopa_M]

Hello,

Thank you everyone for your responses and bringing this issue to our notice. .

I have gone through all the comments and it seems like most of you are affected with this. However, for reporting bugs/features this is not a best suitable forum. Hence I would request you to raise a bug with public issue tracker with required details and screenshots as you can best explain the scenario over there. Our issue tracker team will look into it and accordingly  prioritize your request and provide you with the assistance you need.