alloydb omni 15.2.2 agent error messages regarding failed to install some extensions

[bbakkebo]

Hi I am seeing a lot of errors on a reboot of my alloydbtest server

It seems to be trying to reinstall the agents and such, but fails and is looping.
Is there anything I can do to fix this to get it to stop?  or if it is standard behavior perhaps I can just turn this off in the logging?

2023-12-13 00:45:54.087 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: Operation agent starts reinstalling extension google_db_advisor in database template1#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: ERROR: failed to run query DROP EXTENSION IF EXISTS google_db_advisor: pq: cache lookup failed for relation 18903#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: WARNING: Operation agent failed to reinstall extension google_db_advisor for DB:template1. error: pq: cache lookup failed for relation 18903#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: Operation agent starts reinstalling extension hypopg in database template1#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: ERROR: failed to run query DROP EXTENSION IF EXISTS google_db_advisor: pq: cache lookup failed for relation 18903#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: WARNING: Operation agent failed to reinstall extension google_db_advisor for DB:template1. error: pq: cache lookup failed for relation 18903#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: Operation agent starts reinstalling extension hypopg in database template1#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: ERROR: failed to run query DROP EXTENSION IF EXISTS hypopg: pq: cache lookup failed for relation 18990#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: ERROR: failed to run query DROP EXTENSION IF EXISTS hypopg: pq: cache lookup failed for relation 18990#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: WARNING: Operation agent failed to reinstall extension hypopg for DB:template1. error: pq: cache lookup failed for relation 18990#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: reinstall_extensions_bin failed to install some extensions in some databases. Back off and retry.#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: WARNING: Operation agent failed to reinstall extension hypopg for DB:template1. error: pq: cache lookup failed for relation 18990#015
2023-12-13 00:45:54.089 CET
Dec 12 23:45:54 dptest-alloydb ccb6273a2ac5[984]: reinstall_extensions_bin failed to install some extensions in some databases. Back off and retry.#015
2023-12-13 00:45:57.092 CET
Dec 12 23:45:57 dptest-alloydb ccb6273a2ac5[984]: DSN: dbname='alloydbadmin' host=127.0.0.1 port=5432 user=alloydbadmin sslmode=disable application_name=reinstall_extensions_bin#015
2023-12-13 00:45:57.092 CET
Dec 12 23:45:57 dptest-alloydb ccb6273a2ac5[984]: DSN: dbname='alloydbadmin' host=127.0.0.1 port=5432 user=alloydbadmin sslmode=disable application_name=reinstall_extensions_bin#015
2023-12-13 00:45:57.096 CET
Dec 12 23:45:57 dptest-alloydb ccb6273a2ac5[984]: DSN: dbname='template1' host=127.0.0.1 port=5432 user=alloydbadmin sslmode=disable application_name=reinstall_extensions_bin#015
2023-12-13 00:45:57.096 CET
Dec 12 23:45:57 dptest-alloydb ccb6273a2ac5[984]: Operation agent starts reinstalling extension google_columnar_engine in database template1#015

[ms4446]

These errors suggest that the agent is attempting to reinstall extensions but is encountering issues in dropping existing extensions before installing new ones. This could be due to various reasons, such as database file corruption or incompatibilities between extensions.

To address this issue, I recommend the following steps:

1. In-Depth Log Analysis: Carefully examine the AlloyDB logs for more detailed information about the errors. Pay special attention to messages related to the extensions that are failing to install, as this can offer insights into whether the issue is widespread or specific to certain extensions.

2. Confirm Extension Compatibility: Ensure that the extensions you're attempting to install are compatible with your version of AlloyDB. Additionally, check if these extensions are compatible with each other. This information is typically available in the AlloyDB documentation.

3. Manual Dropping of Extensions: If the agent is struggling to drop the extensions automatically, consider manually removing them using the DROP EXTENSION SQL command. However, this should be done cautiously, preferably by someone with database administration experience. Before proceeding, ensure you have a current backup of your database to safeguard against any potential data loss.

4. Backup and Recovery Preparedness: Emphasize the importance of having a recent backup before making any manual changes to the database. This ensures you can restore the database to its previous state if needed.

5. Disabling Auto-Reinstallation: If the problematic extensions are not essential for your operations, you might consider disabling their auto-reinstallation in the AlloyDB configuration. This is a temporary workaround and should be used with caution. It's important to understand why these extensions are failing to install and to address the underlying cause.

6. System Health Check: Ensure that the system hosting AlloyDB is functioning optimally. System-level issues can sometimes cause database-related errors.

7. Post-Change Monitoring: After implementing any changes, closely monitor the database for new issues or unexpected behaviors to ensure stability.

[bbakkebo]

Hi again,

Thanks for your reply here.
This was a brand new install.   Installed alloydb omni by following quckstart, then added ssl to postgres and upped the server specs.
But strangely enough I see that everything seems to be working normally again. So it fixed it self over time.  Just a quick question here.  Will there be more documentation in the future for Omni in the future?  As of now there is very little and we see that most things are geered to alloydb on google cloud which we are going to be moving our production database over in the near future as things have been working well here.
2023-12-13 22:04:26.172 CET
Check state: resource "install-pkg" state is COMPLIANT.
2023-12-13 22:04:26.172 CET
Policy "goog-ops-agent-policy" resource "add-repo" state: COMPLIANT
2023-12-13 22:04:26.172 CET
Policy "goog-ops-agent-policy" resource "install-pkg" state: COMPLIANT
2023-12-13 22:04:26.172 CET
Dec 13 21:04:26 dptest-alloydb OSConfigAgent[551]: 2023-12-13T21:04:26.1721Z OSConfigAgent Info: Check state: resource "install-pkg" state is COMPLIANT.
2023-12-13 22:04:26.172 CET
Dec 13 21:04:26 dptest-alloydb OSConfigAgent[551]: 2023-12-13T21:04:26.1722Z OSConfigAgent Info: Policy "goog-ops-agent-policy" resource "add-repo" state: COMPLIANT
2023-12-13 22:04:26.172 CET
Dec 13 21:04:26 dptest-alloydb OSConfigAgent[551]: 2023-12-13T21:04:26.1723Z OSConfigAgent Info: Policy "goog-ops-agent-policy" resource "install-pkg" state: COMPLIANT
2023-12-13 22:04:26.261 CET
Successfully completed ApplyConfigTask

[ms4446 · Accepted solution]

I'm glad to hear that the issues with your AlloyDB Omni installation resolved themselves over time. It's not uncommon for systems to experience initial hiccups during setup, especially with complex software like database management systems.

Regarding your question about future documentation for AlloyDB Omni, it's quite typical for new software and services, especially those as complex as database systems, to have limited documentation initially. As the product matures and gains a larger user base, the documentation usually expands and improves. This is often a response to the growing needs and feedback from users like yourself.