Issue with Incomplete Detection of Mismatches Between Documents in Different Languages

[PhuuPwint]

Hello everyone,

I’m encountering an issue while using Google cloud console (Vertex AI studio's Chat section) to compare two documents written in different languages but with the same content. I expect the system to identify all differences or mismatches between the documents, but it only returns 2–3 mismatches in a single response, even though there are more than 5 mismatches in reality.

Here are the details of my situation:

• Input: Two documents with same content but in two different languages.
• Expected Output: A complete list of all mismatches or differences between the documents in a single response.
• Actual Output: Only 2–3 mismatches are identified, while additional mismatches are ignored.
• Attempted Solutions: I have tried with various prompt instructions, 8192-token output limit and gemini 1.5 pro-002.

Is there a way to ensure the system provides a comprehensive list of all mismatches in one response? Am I potentially missing a configuration or parameter setting that affects the output?

Thank you for your help!

[ruthseki]

Hi @PhuuPwint,

Welcome to Google Cloud Community!

I understand that you're trying to get a comparison of two documents in different languages using Vertex AI Studio's Chat section, specifically Gemini 1.5 pro-002, but you're only seeing a partial list of mismatches. 

Here's why your current approach is unlikely to succeed:

• Tokenization limits: Even with an 8192-token limit, exceeding that limit during the comparison process might lead to truncation. The LLM might only process a portion of the documents, leading to an incomplete analysis.
• Context window limitations: The model's "context window" (the amount of text it can actively consider at once) is limited. Comparing very large documents might exceed this limit, forcing the model to process them in chunks, potentially missing connections between different parts of the text.

You may try these approaches:

1. Ensure that the model parameters are set correctly. Sometimes, adjusting parameters like temperature can affect the completeness of the output. While you haven't mentioned this, the model's temperature parameter controls the randomness of responses. A lower temperature (closer to 0) might lead to more consistent results. You could try a lower temperature if you're getting wildly inconsistent results between attempts.

2. Review your prompt design. Make sure it clearly instructs the model to list all mismatches. You might want to experiment with different phrasings to see if they yield better results.

3. If the documents are long, consider breaking them into smaller sections and processing them in batches. This might help the model handle the comparison more effectively. You may refer to this documentation for reference.

4. Verify that the model you're using supports both languages you're working with. Some models might have better support for certain languages than others. In addition, you've specified using gemini 1.5 pro-002. Note that newer models will likely be better at this task. Ensure you're using the most up-to-date model version, if possible.

5. Use the feedback mechanism in Vertex AI to report this issue. Google's team might provide insights or updates to improve the functionality.

Here’s a sample documentation that you may find useful:

*Document Comparison

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[PhuuPwint]

Dear @ruthseki,

Thank you for providing an explanation regarding my issue.

To summarize, can I confirm that the issue may likely be related to the context window and token limit, despite the context window size being 2M and the output token limit of the model being 8192?

Based on your suggestions, here are the approaches I have already tried:

1. I have set the temperature to 0.
2. I provided clear instructions using prompt engineering techniques, ensuring the prompt explicitly instructed the model to list all mismatches.
3. I included instructions in the prompt to process the text in smaller chunks. However, I am unsure how to implement batch processing within the chat section of the Vertex AI console without using code.
4. I tested the documents in two languages, English and Bahasa Malay (BM). During this testing, I discovered that the model I used does not support BM. Could this be a potential cause of the issue?
5. I have already submitted the feedback form, as you suggested. Thank you for mentioning this.

One thing is the documents I tested are up to 9 pages of PDF.

I appreciate your assistance so far. Could you kindly provide further guidance regarding my concerns outlined above?

Thank you once again for your help.

Phuu Pwint