Sadly, I'm not a Vertex AI Image specialist ... but I'll throw in my 2 cents. When we perform text extraction from an image, we get back a structured record describing what was found. I believe that this is the doc. When we look at a record, we see that each entity has an associated "score" between 0 and 1 that represents the confidence/score. If the score is high, Vertex AI is claiming that it is more confident in the result ... if low, then Vertex AI has less confidence in the result. You started your post with the notion that Vertex AI is correct 99.9% of the time. From my perspective, that feels good. We usually think of software as being correct 100% of the time ... 1 + 1 had always better equal exactly 2 ... however I'm getting the sense that in your design, you are "grabbing" screen images, converting them to image format and then passing those image formats through Vertex AI to extract information. I liken this to the notion of instead of sending someone a spreadsheet of data, we are printing out the spreadsheet, taking a photocopy, sending it via fax and then asking someone to re-enter the data. Transcription errors will occur.
In Vertex AI, the score value is typically used to direct extracted data to a secondary process for resolution. For example, if this were an "invoice" being processed, we had better be sure that the amounts are as accurate as possible. If a recognition score were low, we might send that to a human to review and correct if necessary while allowing good scored data (say 97% or above) to be passed through without human interaction.
LinkedIn
Twitter
