Cloud SQL Major Version Upgrade Failure: Error Code 1005

[gateweb]

HI 

We attempted to perform a major version upgrade for our Cloud SQL instance, but the operation failed with the error code 1005. Below are the details of the failed operation:

1. Operation Details:

   • Service Name: cloudsql.googleapis.com
   • Method Name: cloudsql.instances.update
   • Operation ID: 13b52dec-2faf-4a40-92fb-a86300000025
   • Timestamp: 2024-11-28T06:47:37.833398Z

2. Upgrade Attempt:

   • Current Database Version: POSTGRES_9_6
   • Target Database Version: POSTGRES_14
   • Intent: Major Version Upgrade (MAJOR_VERSION_UPGRADE)

3. Error Details:

   • Status Code: 1005
   • Severity: ERROR

4. Additional Context:

   • This operation was performed via the Cloud SQL Admin API.
   • The operation ID indicates that this was the last attempt related to this update, but no further details about the root cause of the failure are provided in the logs.

---

Assistance Requested:

We kindly request your assistance with the following:

1. Error Explanation:

   • Could you clarify what error code 1005 signifies in this context?
   • Is there any specific compatibility or configuration issue preventing the upgrade from POSTGRES_9_6 to POSTGRES_14?

2. Debugging Details:

   • Can you provide more details about why this operation failed?
   • Is there any action required on our side to make this upgrade successful?

3. Recommended Next Steps:

   • Are there alternative approaches or upgrade paths we should follow (e.g., intermediate version upgrades)?
   • Do we need to perform additional pre-checks or make configuration adjustments?

4. Logs and Troubleshooting:

   • If more logs or operation details are required to troubleshoot, please let us know how to provide them.
     
     
     
5. Json log

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 1005
    },
    "serviceName": "cloudsql.googleapis.com",
    "methodName": "cloudsql.instances.update",
    "metadata": {
      "@type": "type.googleapis.com/speckle.UpdateIntentEventLog",
      "intents": [
        {
          "diffs": [
            {
              "newValue": "POSTGRES_14",
              "oldValue": "POSTGRES_9_6",
              "property": "database_version"
            }
          ],
          "intent": "MAJOR_VERSION_UPGRADE"
        }
      ]
    }
  },
  "insertId": "7a1hr7e61pmg",
  "timestamp": "2024-11-28T06:47:37.833398Z",
  "severity": "ERROR",
  "operation": {
    "id": "13b52dec-2faf-4a40-92fb-a86300000025",
    "producer": "cloudsql.googleapis.com",
    "last": true
  },
  "receiveTimestamp": "2024-11-28T06:47:38.062535257Z"
}

Additional Notes:

• This instance is cloned from a production environment instance.
• We also attempted upgrading to a closer version (POSTGRES_10), but encountered the same error (1005).
• The instance uses SSD storage with a size of 700 GB. If we were to use pg_dump and pg_restore to migrate to a new instance, it would require significant time, during which the system might not be able to operate.
• We would prefer to complete this upgrade in place via GCP’s built-in upgrade process due to time and operational constraints.
• The same database setup exists in AWS, where the automatic upgrade process to the desired version has been completed successfully.
• Due to the cloned nature of this instance, it might incur higher costs. As a result, we have stopped the instance for now. Please notify us if it needs to be restarted for investigation.
• However, the extension of GCP’s support for this version is nearing its deadline, so we would greatly appreciate clarification on why the automatic upgrade is failing and any guidance on resolving this issue.

Thank you for your assistance!

[mcbsalceda]

Hi @gateweb,

Unfortunately, error code 1005 in Google Cloud Platform (GCP) is generic and signifies an internal error. It doesn't provide specific details about the issue.  While there's no confirmation, a significant version jump (PostgreSQL 9.6 to 14) might have compatibility issues that the upgrade process cannot handle automatically. Upgrading to a closer version (10) failing with the same error suggests this possibility. 

For a full guide on how to perform an in-place major version upgrade, check out the documentation here. It's a good idea to get in touch with Google Cloud Support. They can access internal logs and have expertise to diagnose the root cause of error 1005 in this specific context. When reaching out, be sure to include as much detail as possible, including any screenshots, to help them understand the issue and resolve it faster.

Hope this helps.