From some weeks, we successfully created some Vertex AI Search Indexes in the console UI and with Terraform in several projects, some creations lasted around 30 mins, some even only 3 or 4 mins.
From yesterday 27/11, all our Indexes creation are never ending, staying with the 'Creating' status for hours (even after more than 4 hours for one of them).
Example of a test in the console UI:
- region: europe-west3
- algorithm type: TreeAH algorithm
- dimensions: 768
- Approximate neighbors count: 150
- update method: Stream
- shard size: Small
- distance measure type: Dot product distance
- feature norm type: None
- fraction leaf nodes to search: 0.05
- lean node embedding count: 1000
Example of a test with Terraform:
resource "google_vertex_ai_index" "vertexai_index" {
project = module.target_project.project_id
region = "europe-west3"
display_name = "vertexai-index"
description = "vertexai-index"
metadata {
contents_delta_uri = "gs://${google_storage_bucket.vertexai_index_bucket.name}/contents"
config {
dimensions = 768
approximate_neighbors_count = 150
shard_size = "SHARD_SIZE_SMALL"
distance_measure_type = "DOT_PRODUCT_DISTANCE"
algorithm_config {
tree_ah_config {
leaf_node_embedding_count = 500
leaf_nodes_to_search_percent = 7
}
}
}
}
index_update_method = "STREAM_UPDATE"
depends_on = [
google_storage_bucket_object.data
]
}
What we have tested / checked so far:
1. Tried creating 5 or 6 indexes with europe-west3 or us-central1
2. confirmed in the documentation that creation can take up to 1 hour:
https://cloud.google.com/vertex-ai/docs/vector-search/create-manage-index
"Your new index appears in your list of indexes once it's ready. Note: Build time can take up to an hour to complete."
However we are far from 1 hour here, even after 4 / 5 hours still not created
3. checked permissions, the terraform service account and the UI users have been granted Owner for troubleshooting reasons, still the issue was there
4. quotas on the newly created project seem ok (everything green)
Vertex AI API / Matching Engine concurrent index creation operations. per region: 1/ 5
Vertex AI API / Matching Engine Indexes per region: 1 / 15
5. everything green for quotas at organization level
6. No error found in Cloud Logging, only the index never created, status keeps showing Creating after some hours (more than 4 hours for an index creation today and still not created)
7. gcloud ai indexes list or describe <index id> --region=<region> : return "0 items" and "NOT_FOUND: Index `projects/<project id>/locations/europe-west3/indexes/<index id>` is not found
We do not know what else to check:
- There is no list method for "gcloud ai operations", we do not know how to get the operations id and try to get some clarity with the describe command
(https://cloud.google.com/sdk/gcloud/reference/ai/operations)
- not sure if using another dimensions number or approximate neighbors count or any other parameter would change something
Would someone have a clue ?
LinkedIn
Twitter
