Entity Search Using Gemini in Agents Builder - Retrieval Step Issues

[pedropcamellon]

I'm currently working on a project where I need to perform entity searches using Gemini in Agents Builder, specifically for company names. However, I'm struggling with the retrieval step and would appreciate any insights or suggestions you might have.

I'm particularly concerned about the ability of vector stores to effectively match company names or usernames when only an additional field like country (where the company is based) is available. I'm wondering if this limitation might be the root cause of my retrieval issues. Has anyone successfully implemented a similar setup where company names are matched accurately using vector stores with such limited context? If so, how did you overcome the challenges of disambiguating similar company names across different countries?

Here's a brief overview of my setup:

1. **Data Preparation:**
- I export company names and country codes from my SQL database as JSON files.
- Due to constraints, I change the file extension to `.txt` before uploading them to the data store.

An example of the data structure in these files is as follows:

```json
{
"companyName": "Heaths Paint Center INC",
"countryCode": "USA"
},
{
"companyName": "Heath W Holtapp",
"countryCode": "USA"
}
```

2. **Agent Instructions:**
- The agent is instructed to search for a company based on the provided name and country code.
- It capitalizes each word in the company name and uses a tool to search for the most similar name in the records.
- If a match is found, it returns the company name and the source in JSON format.

Despite this setup, the retrieval step often fails to return results for company names that are only slightly different from those in the data store. I am encountering inconsistent results. Sometimes, even when I use the exact same company names as those in the data store, the retrieval step fails to return any results. This inconsistency is perplexing, especially for such simple tests where I expect an exact match.

I have tried normalizing the company names by converting them to lowercase, removing punctuation, and standardizing common suffixes like "Inc." and "Ltd." before indexing them. Despite these efforts, the retrieval still fails frequently.

I was expecting the retrieval step to consistently return matches for exact or near-exact company names, especially given that the names are identical in some tests. This inconsistency suggests there might be an underlying issue with how the vector stores handle the matching, particularly when only an additional field like the country code is provided.

Here are some specific questions and issues I'd like to address:

1. **Fuzzy Matching:**
- Is there a way to enable or improve fuzzy matching in the retrieval step to account for small variations in company names?
- How can I ensure that similar names, even with slight differences, are matched correctly?

2. **Vector Stores and Contextual Matching:**
- Does Gemini support vector stores for matching company names or usernames with additional fields like country codes?
- If so, how can I optimize the vector store setup to improve retrieval accuracy?

3. **Data Normalization and Preprocessing:**
- What are the best practices for normalizing and preprocessing company names before indexing them in the data store?
- Are there any specific techniques or tools recommended for handling common variations in company names (e.g., "Inc.", "Ltd.")?

4. **Handling Large Datasets:**
- Are there any performance considerations or limitations when working with large datasets in this context?
- How can I ensure efficient and accurate retrieval from a large data store?

Any advice, examples, or resources you could provide would be greatly appreciated. I'm particularly interested in hearing from anyone who has successfully implemented a similar setup or faced and resolved similar issues.

Thank you in advance for your help!

[McMaco · Accepted solution]

Hi pedropcamellon,

Welcome to Google Cloud Community!

For your reference regarding Entity Search Using Gemini in Agents Builder - Retrieval Step Issues you may visit this documentation supporting all items with regards to Gemini for Google Cloud. 

1. **Fuzzy Matching**

Unfortunately, Agents Builder itself doesn't currently offer built-in fuzzy matching functionalities within the retrieval step for Gemini. However, there are several strategies you can employ to improve retrieval accuracy for company names with slight variations:

Preprocessing Text:

• Standardize Text: Before feeding company names to Gemini, standardize them using techniques like:
• Lowercasing all characters
• Removing punctuation and special characters
• Standardizing abbreviations (e.g., "Inc." to "Incorporated")

This reduces inconsistencies and improves the quality of the vector representations used for matching.

Data Enrichment:

• Expand Entity Data: Go beyond just company name and country. Include additional information in your entities to provide richer context for differentiation. Consider adding:
• Full company addresses
• Industry codes

This additional data helps create more distinct vectors, leading to better retrieval of similar names.

While Agents Builder lacks built-in fuzzy matching, there are workarounds:

• Synonym Expansion: Create synonym lists for common company name variations (e.g., "Google LLC" with synonyms like "Google", "Google Corporation").
• External Libraries: Integrate external libraries like FuzzyWuzzy (Python) or JaroWinkler (various languages) for preprocessing before feeding data to Gemini. These libraries identify similar names based on edit distance algorithms.

2. **Vector Stores and Contextual Matching**

Yes, Gemini can be used with vector stores for matching company names or usernames even when you have additional fields like country codes. You may refer to input data format and structure for more information.

3. **Data Normalization and Preprocessing**

Here are some best practices for normalizing and preprocessing company names before indexing them in the data store for use with entity search using Gemini in Agents Builder

Stop Word Removal (Optional): Depending on your data and desired outcome, consider removing common stop words (e.g., "the", "a", "an") that don't hold much meaning for company names. However, be cautious as some stop words might be relevant for specific companies (e.g., "The North Face").

Stemming or Lemmatization (Optional): Explore techniques like stemming (reducing words to their base form) or lemmatization (converting words to their dictionary form). This can help capture variations like "running" and "ran" as the same concept. However, evaluate the impact on accuracy, as stemming might lead to unintended word reductions (e.g., "access" becoming "ac"). Lemmatization might be more reliable.

Specific Techniques:

• Synonym Lists: Create synonym lists that map variations of company names to their canonical forms. For instance, the list for "Google LLC" might include synonyms like "Google", "Google Corporation", and "Alphabet Inc." (Google's parent company). During preprocessing, you can replace variations with their corresponding canonical form in the synonym list.
• Regular Expressions: Utilize regular expressions to identify and replace specific patterns in company names. For example, a regular expression could replace all occurrences of "(Inc.|Ltd.|Corp.)$" (end of string match for Inc., Ltd., or Corp.) with an empty string, effectively removing these suffixes.

4. **Handling Large Datasets**

Ensuring efficient and accurate retrieval from a large data store, especially for entity searches with company names in Agents Builder and Gemini, requires a multi-pronged approach.

Remember, the most effective approach might involve a combination of these techniques, tailored to the specific size and characteristics of your data.

I hope the above information is helpful.

[pedropcamellon]

Hi!
Thank you for your detailed response. I'm glad to hear that you found the suggestion about data enrichment to be truly important. Your insight about expanding entity data beyond just company name and country is spot on. Adding more context, such as full company addresses and industry codes, can indeed significantly improve the differentiation and richness of the data.

Thank you for your detailed response. I appreciate the insights, especially regarding the use of vector stores for contextual matching in Vertex AI.

However, I found this section a bit confusing:

```markdown
2. **Vector Stores and Contextual Matching**

Yes, Gemini can be used with vector stores for matching company names or usernames even when you have additional fields like country codes. You may refer to input data format and structure for more information.
```

Across Vertex AI and the Agents Builder, the terminology and feature names can be quite messy, leading to a lot of confusion. I often found myself lost in the documentation. My suggestion is to unify the features or name them more clearly. In this context, you referenced a feature in Vertex AI while I am actually using data stores—a managed service that creates and serves embeddings for me. Clearer naming and documentation would greatly enhance usability and reduce confusion.

Thanks again for your help!

Citations:
[1] https://cloud.google.com/vertex-ai/docs/vector-search/overview
[2] https://python.langchain.com/v0.2/docs/integrations/vectorstores/google_vertex_ai_vector_search/
[3] https://pub.dev/documentation/langchain_google/latest/langchain_google/VertexAIMatchingEngine-class....
[4] https://news.clateway.com/new-vertex-ai-feature-store-built-with-bigquery-ready-for-predictive-and-g...
[5] https://cloud.google.com/blog/products/ai-machine-learning/new-vertex-ai-feature-store-bigquery-powe...

[6] Data stores  |  Vertex AI Agents  |  Google Cloud