From some weeks, we successfully created some Vertex AI Search Indexes in the console UI and with Terraform in several projects, some creations lasted around 30 mins, some even only 3 or 4 mins.
From yesterday 27/11, all our Indexes creation are never ending, staying with the 'Creating' status for hours (even after more than 4 hours for one of them).
Example of a test in the console UI:
- region: europe-west3
- algorithm type: TreeAH algorithm
- dimensions: 768
- Approximate neighbors count: 150
- update method: Stream
- shard size: Small
- distance measure type: Dot product distance
- feature norm type: None
- fraction leaf nodes to search: 0.05
- lean node embedding count: 1000
Example of a test with Terraform:
resource "google_vertex_ai_index" "vertexai_index" {
project = module.target_project.project_id
region = "europe-west3"
display_name = "vertexai-index"
description = "vertexai-index"
metadata {
contents_delta_uri = "gs://${google_storage_bucket.vertexai_index_bucket.name}/contents"
config {
dimensions = 768
approximate_neighbors_count = 150
shard_size = "SHARD_SIZE_SMALL"
distance_measure_type = "DOT_PRODUCT_DISTANCE"
algorithm_config {
tree_ah_config {
leaf_node_embedding_count = 500
leaf_nodes_to_search_percent = 7
}
}
}
}
index_update_method = "STREAM_UPDATE"
depends_on = [
google_storage_bucket_object.data
]
}
What we have tested / checked so far:
1. Tried creating 5 or 6 indexes with europe-west3 or us-central1
2. confirmed in the documentation that creation can take up to 1 hour:
https://cloud.google.com/vertex-ai/docs/vector-search/create-manage-index
"Your new index appears in your list of indexes once it's ready. Note: Build time can take up to an hour to complete."
However we are far from 1 hour here, even after 4 / 5 hours still not created
3. checked permissions, the terraform service account and the UI users have been granted Owner for troubleshooting reasons, still the issue was there
4. quotas on the newly created project seem ok (everything green)
Vertex AI API / Matching Engine concurrent index creation operations. per region: 1/ 5
Vertex AI API / Matching Engine Indexes per region: 1 / 15
5. everything green for quotas at organization level
6. No error found in Cloud Logging, only the index never created, status keeps showing Creating after some hours (more than 4 hours for an index creation today and still not created)
7. gcloud ai indexes list or describe <index id> --region=<region> : return "0 items" and "NOT_FOUND: Index `projects/<project id>/locations/europe-west3/indexes/<index id>` is not found
We do not know what else to check:
- There is no list method for "gcloud ai operations", we do not know how to get the operations id and try to get some clarity with the describe command
(https://cloud.google.com/sdk/gcloud/reference/ai/operations)
- not sure if using another dimensions number or approximate neighbors count or any other parameter would change something
Would someone have a clue ?
ok guys, it seems that in fact the index creation time is crazy from yesterday
There was no error, only a huge huge creation time.
Created 3 indexes for troubleshooting reason today, and this time kept the creation going to the end.
Creation times were:
- index 1: 7 hours 42 mins,
- index 2: 4 hours 45 mins,
- index 3: 4 hours 43 mins
And we are only talking about creating the indexes, we still have to deploy them on endpoints and to ingest the vectors 😉
We have created a new index yesterday, creation time was again huge, around 8h00.
Could someone from Google check if there is an ongoing incident ?
The documentation states:
"Your new index appears in your list of indexes once it's ready. Note: Build time can take up to an hour to complete."
Same here.
How many entries did your indices have? And which document format did you use for data?
This is only the creation time for the index resource itself with a json file containing one line only, before ingesting data programmatically later.
Hey!
I am also encountering the same problem. I am creating an index with only a few tens of vectors to do some experimentation. Yesterday, I attempted to create an index and it took a really long time (I left it overnight) for the status to change from creating to something else.
Today, I created a new index about 2 hours ago, and it is still going.
Jurgen
The creation succeeded for me after running for something more than 7 hours.
@luda Hey , Can you help me on how you are checking the status of index creation. I am not able to find it in documentation.
I am creating empty index so i get index name in return. Then I am updating it with embeddings data.
Index creation happening in background successfully , I need to check the status through Python code.
Can you please share.
Thank you in advance
Hi amitagarg22,
The idea here is to check the status of the Operation returned by the create_index, update_index, deploy_index methods.
Refer to the "Poll the operation" comments in this page: matching_engine_for_indexing.ipynb
Please note this code is old now (9 months ago), it might be good to check if there are more recent ways to use the indexes.
Hope this helps,
Ludovic