Vertex AI not understanding invoices?

[arimakidd]

I ran a test in Vertex AI as I am excited to start working this new AI.  I submitted two invoices.  There is a specific text in the invoice, "Balance Due".  So I submitted two and asked the following:

"Here are two invoices.  What is the total balance due of all two invoices?"

The response was the total balance due of only 'one' invoice. What can I do to correct this error?  Below is  a screenshot of the attempt.

[OrangiaNebula]

I am not addressing your primary concern directly. But this is valid advice. Your approach has the model doing arithmetic to add the balance from the two invoices. LLMs are not good at arithmetic. They make a good guess what a sum will be and sometimes even come up with the right sum, but they are not good at doing calculations.

Ask the LLM to produce JSON formatting for the invoice data. You will have clean data to work with and can then do the arithmetic in your PC or even mobile device, both of which are better at calculating than the best LLM is.

[arimakidd]

@OrangiaNebula I agree entirely on your advice.  However, my expectations are based on what I saw having attended Google I/O online.  All that had to be done was to deposit documents of the same layout and nature(even different layouts and nature) and the Gemini could calculate a mathematical sum.  There were several instances where this functionality was demonstrated.  It was summarized as 'any input, any output'.  I am looking to leverage on this demonstrated functionality.  Am I using the wrong model?