Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Connecting to Cloud SQL In Cloud Build Through VPC Private Connection

I am trying to connect to a Cloud SQL database during a Cloud Build step so I can run database migrations for my app. But I keep running into errors such as: 

SQLSTATE[HY000] [2002] Operation timed out

I have followed the instructions here very closely:

https://cloud.google.com/sql/docs/mysql/connect-build#php_2

  • I am able to connect to the database through the Cloud SQL Proxy using a public IP address just fine. I would like to switch to private for stronger security so we may disable the public connection.
  • I have a VPC network set up. It's not the default network. I have my Cloud SQL Instance connected to this network. I am able to connect to my Cloud SQL Instance using a private connection through BigQuery, Cloud Run, and even Cloud Functions but not Cloud Build for some reason.
  • I have set up a private connection between the VPC network of my Cloud SQL Instance and the Service Producer network. I have allocated an IP range and made a private connection. I am fairly certain this step is done correctly but I'm still rather new to networks with GCP.
  • I created a Cloud Build Worker Pool. I have tried both a private pool in the VPC network and a private pool in the Service Producer network (when you do not provide a project ID/name or VPC network). Both options produced the same results.
  • I have tried connecting to the database through the Cloud Build step inside the cloudbuild.yaml itself and from a separate file as seen in the documentation, they both produce the same error.
  • I have tried creating and assigning a Service Account to the Cloud Build Trigger and have assigned it all of the proper permissions needed to connect to the Cloud SQL instance. I got the same error.
  • I have given my built-in Cloud Build service account permissions. This service account is in the same project as the VPC and the Cloud SQL instance. Whenever I tried to assign a role it just give me a message saying "No changes were applied, this account already has those permissions".
  • My app is build in Laravel and PHP. I use  cloudbuild.yaml for all the cloud build steps and I have a Dockerfile for installing all the necessary dependencies for the app.
  • I am fairly certain this is a network issue. I have tried random solutions such as making Firewall rules but nothing works. I have tried using a Virtual Machine but am thinking that is unnecessary because Cloud Build already runs in a VM by default. After viewing the documentation, it does not seem that you need a VM at all for this. 

Here is my cloudbuild.yaml:

steps:
  - name: gcr.io/cloud-builders/docker
    args:
      - build
      - '--no-cache'
      - '-t'
      - '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
      - .
      - '-f'
      - Dockerfile
    id: Build
  - name: gcr.io/cloud-builders/docker
    args:
      - push
      - '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
    id: Push
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
    args:
      - run
      - services
      - update
      - $_SERVICE_NAME
      - '--platform=managed'
      - '--image=$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
      - >-
        --labels=managed-by=gcp-cloud-build-deploy-cloud-run,commit-sha=$COMMIT_SHA,gcb-build-id=$BUILD_ID,gcb-trigger-id=$_TRIGGER_ID,$_LABELS
      - '--region=$_DEPLOY_REGION'
      - '--quiet'
    id: Deploy
    entrypoint: gcloud
  # This step fails to make a connection.
  - id: "Connect"
    name: "$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA"
    dir: sql-private-pool
    env:
      - "CLOUD_BUILD_DB_NAME=$_DB_DATABASE"
    secretEnv: ['CLOUD_BUILD_DB_USERNAME', 'CLOUD_BUILD_DB_PASSWORD', 'CLOUD_BUILD_DB_HOST']
    entrypoint: php
    args: ["/app/sql-private-pool/migrate.php"]
  # This step also fails to make a connection.
  - name: 'gcr.io/google-appengine/exec-wrapper'
    entrypoint: 'bash'
    args:
      - -c
      - |
        /buildstep/execute.sh \
          -i $_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA \
          -e DB_CONNECTION=mysql \
          -e DB_HOST=$$CLOUD_BUILD_DB_HOST \
          -e DB_PORT=3306 \
          -e CLOUD_SQL_CONNECTION_NAME=$_DB_CONNECTION \
          -e DB_DATABASE=$_DB_DATABASE \
          -e DB_USERNAME=$$CLOUD_BUILD_DB_USERNAME \
          -e DB_PASSWORD=$$CLOUD_BUILD_DB_PASSWORD \
          -s $_DB_CONNECTION \
          -- php /app/artisan migrate --force
    secretEnv: ['CLOUD_BUILD_DB_USERNAME', 'CLOUD_BUILD_DB_PASSWORD', 'CLOUD_BUILD_DB_HOST']
    id: Migrate
timeout: 1200s
images:
  - '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
options:
  pool:
    name: projects/$PROJECT_ID/locations/$WORKERPOOL_LOCATION/workerPools/$VPC_NAME
  substitutionOption: ALLOW_LOOSE
availableSecrets:
  secretManager:
    - versionName: projects/$PROJECT_ID/secrets/CLOUD_BUILD_DB_PASSWORD/versions/latest
      env: 'CLOUD_BUILD_DB_PASSWORD'
    - versionName: projects/$PROJECT_ID/secrets/CLOUD_BUILD_DB_USERNAME/versions/latest
      env: 'CLOUD_BUILD_DB_USERNAME'
    - versionName: projects/$PROJECT_ID/secrets/CLOUD_BUILD_DB_HOST/versions/latest
      env: 'CLOUD_BUILD_DB_HOST'
tags:
  - gcp-cloud-build-deploy-cloud-run
  - gcp-cloud-build-deploy-cloud-run-managed
 
 
Here is my migrate.php:
 
<?php

namespace Google\Cloud\Samples\CloudSQL\MySQL;

use PDO;
use PDOException;
use RuntimeException;
use TypeError;

class DatabaseTcp
{
    public static function initTcpDatabaseConnection(): PDO
    {
        try {
            $username = getenv('CLOUD_BUILD_DB_USERNAME');
            $password = getenv('CLOUD_BUILD_DB_PASSWORD');
            $dbName = getenv('CLOUD_BUILD_DB_NAME');
            $instanceHost = getenv('CLOUD_BUILD_DB_HOST');

            // Connect using TCP
            $dsn = sprintf('mysql:dbname=%s;host=%s', $dbName, $instanceHost);

            // Connect to the database
            $conn = new PDO(
                $dsn,
                $username,
                $password,
                [
                    PDO::ATTR_TIMEOUT => 5,
                    PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
                ]
            );

            // Check if the connection is successful
            if ($conn->isConnected()) {
                print("\n");
                print("=========================");
                print("\n");
                print("Connected to the database successfully!");
            } else {
                print("\n");
                print("=========================");
                print("\n");
                print("Failed to connect to the database.");
            }


            // Run migration command
            exec('cd .. && php /app/artisan migrate --force', $output, $returnVar);
            if ($returnVar !== 0) {
                throw new RuntimeException('Migration failed: ' . implode("\n", $output));
            }
        } catch (TypeError $e) {
            throw new RuntimeException(
                sprintf(
                    'Invalid or missing configuration! Make sure you have set ' .
                        '$username, $password, $dbName, and $instanceHost (for TCP mode). ' .
                        'The PHP error was %s',
                    $e->getMessage()
                ),
                $e->getCode(),
                $e
            );
        } catch (PDOException $e) {
            print($e->getMessage());
            throw new RuntimeException(
                sprintf(
                    'Could not connect to the Cloud SQL Database. Check that ' .
                        'your username and password are correct, that the Cloud SQL ' .
                        'proxy is running, and that the database exists and is ready ' .
                        'for use. For more assistance, refer to %s. The PDO error was %s',
                    $e->getMessage()
                ),
                $e->getCode(),
                $e
            );
        }

        return $conn;
    }
}

try {
    DatabaseTcp::initTcpDatabaseConnection();
    print("\n");
    print("=========================");
    print("\n");
    print('Connected to Cloud SQL');
} catch (RuntimeException $e) {
    print("\n");
    print("=========================");
    print("\n");
    print('Error: could not connect to Cloud SQL!');
}
0 10 4,010
10 REPLIES 10

Hi @vincemcgaj,

Welcome to the Google Cloud Community!

You can try the following troubleshooting options:

In Configure Cloud Build, under the Private IP tab, Cloud Build must be in the same VPC network as your Cloud SQL Instance. To configure this:

  1. Set up a private connection between the VPC network of your Cloud SQL instance and the service producer network.
  2. Create a Cloud Build Private PoolOnce configured, your application will be able to connect directly using your instance's private IP address and port 3306 when your build is run in the pool.

You should also take a look at this Stack Overflow Post as you might have the same problem. Also, according to this post, Cloud Build does not support VPC networks, therefore it is not possible to connect to the private IP of the Cloud instance. Which contradicts the above option.

I suggest contacting Google Cloud Support to further look into your case. Let me know if it helped, thanks!

 

@Marramirez Thank you for the reply. It would make sense if you cannot connect to a Cloud SQL database from Cloud Build through a private IP because I am fairly certain I have done everything right and followed all those steps listed and I am able to connect through other service such as Cloud Run or BigQuery. However, the documentation provided by Google suggests otherwise because you have a Public IP option and a Private IP options. I'm not fully satisfied by that stack overflow post because as you said the documentation contradicts this and also why would there be Cloud Build Worker Pools that connect to a VPC if Cloud Build does not support VPC? That is confusing.

I noticed with BigQuery that I was able to make a private connection by just providing the connection name and no IP address. Should I be connecting through the DB_SOCKET? I thought socket connections did not work for private connections? Whenever I tried to connect a Cloud function or a Cloud Run service to Cloud SQL it requires the private IP. But BigQuery required the connection name instead. I have tried connecting through DB_SOCKET as well before, no luck.

 

Hi @vincemcgaj , were you able to get this issue resolved. I am having a similar issue connecting to cloud SQL from Cloud build.

@milindajuan unfortunately, no I have not found a solution. I just gave up and switched to using a public connection which works just fine. That's what I suggest. Public connection is still pretty secure and you do not need to add an IP address to the Cloud SQL connection list for the Cloud Build. Just leave the connections list on your Cloud SQL connections empty and no one can publicly connect.

Good luck if you are going to try this. I have already wasted a lot of time trying and I don't think you can actually connect to Cloud SQL from Cloud Build through VPC connection. So my warning is don't waste time on this and just use a public connection.

Hey @vincemcgaj , I was able to resolve the issue with the cloud build workerPool.  Once we create the worker pool and run the build in there, it connects without an issue. Thank you for your respopnse.

@milindajuan Really? I did that as well, connected using a worker pool and it never worked. Either I'm doing something wrong or Google fixed something to make this possible. I'm probably just doing something wrong, oh well. Feel free to share your code, it may become very helpful to someone in the future.

Basically this is my test image - for anyone with future need.

 

steps:
  # build the container image
  - id: 'build'
    name: 'docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/image-x', '.']

  - id: 'push'
    name: 'docker'
    args: ['push', 'gcr.io/$PROJECT_ID/image-x']

  # Running Tests
  - id: 'test'
    name: 'gcr.io/$PROJECT_ID/image-x'
    entrypoint: 'bash'
    env:
      - 'DATABASE_URL_TEST=sqlite:///:memory:'
    secretEnv:
      - DATABASE_URL
    args:
      - '-c'
      - |
        chmod +x bin/console        
        composer test:build
        composer build:database

    # Deploy container image to Cloud Run
  - id: 'deploy'
    name: 'gcr.io/cloud-builders/gcloud'
    args: ['beta', 'run', 'deploy', 'image-x', '--image', 'gcr.io/$PROJECT_ID/image-x', '--region', 'australia-southeast1', '--platform', 'managed','--allow-unauthenticated']

options:
  pool:
    name: projects/$PROJECT_ID/locations/australia-southeast1/workerPools/sql-connector

availableSecrets:
  secretManager:
    - versionName: projects/$PROJECT_ID/secrets/DATABASE_URL/versions/latest
      env: DATABASE_URL

 

 

Thank you for the reply. It looks like you are connecting to a sqlite database for unit test purposes. I'm not trying to do that, I'm trying to connect to a Cloud SQL database so we can run database migrations on the production MySQL database when deploying newer versions of our app. I do not think this will work for me. But thank you for trying.

Nope. You understood incorrectly. If you look closer there are 2 variables. One for the test and the other for the actual database. Sqllite is to run integration tests and the actual connection DATABASE_URL is to run the migrations in the cloud SQL.

@milindajuan Interesting. I'm assuming you are running the migration in 

composer build:database

 I've never seen it done like this. I might try this myself but there are no commands named build in my namespace. So would we have to define this method ourselves?