Discrepancy in Field Extraction: Fine-Tuned Processor UI vs. API Output

[andressasoares]

I have a fine-tuned processor that achieves 1.00 precision for a specific field. When I import a test file of a document using this version, it extracts the field correctly. However, when I use the API to extract data from the same document, using the same version, it extracts a different value for that field. Does anyone know what might be causing this issue and could help me troubleshoot?

[ruthseki · Accepted solution]

Hi @andressasoares,

Welcome to Google Cloud Community!

It appears there's a discrepancy between the results obtained from local file processing and API processing, despite utilizing the same fine-tuned processor version.

Here are some approaches that you may try:

1. Verify Identical Input:

• Pre-processing differences:
• File Format: Double-check that the API is receiving the exact same file format and content as your local test. For instance, if your local test uses a text file but the API receives a PDF, or if there are differences in compression (e.g., different PDF optimization), the OCR process might produce different results.
• Character Encoding: Ensure the character encoding is consistent between your local environment and the API request. Mismatched encoding can lead to text interpretation differences.
• Image Resolution: If the input includes images (e.g., scanned PDFs), differences in the image resolution or quality can affect OCR accuracy. Make sure the API is receiving the same image fidelity.
• OCR Settings: Although you're using the same model, the specific OCR engine configuration on the API server might be slightly different from your local setup. This is especially relevant if you're doing OCR locally and the API does it on the server-side.
• Payload Differences:
• API Request Headers: Scrutinize the headers you are sending with the API request (e.g., Content-Type, language specification). These can subtly influence how the API processes the input.
• API Request Parameters: Review any parameters you're passing in your API request. Are you potentially overriding any default behavior or influencing how the field extraction is handled?
• API Input Format: Confirm the API expects the input in the exact format you are providing (e.g., base64 encoded file contents, multipart form data).

2. Inspect API Output:

• Raw API Response: Instead of just looking at the extracted field, check the complete raw API response. This could contain valuable information like:
• Confidence Scores: Inspect the confidence scores associated with the field extraction. A lower confidence might indicate a more tentative extraction.
• Underlying Data: Look for any underlying text or structured data that was generated during the processing. This could help you understand why a different value was extracted.
• Logs/Debug Information: See if the API provides any debug or log information about its processing. This can sometimes reveal subtle issues.

3. Examine the Processor Itself:

• Model Version: Double-check that the API is actually using the exact same version of your fine-tuned model. A small version number mismatch can cause unpredictable results.
• Processor Logic: If your "processor" involves any preprocessing steps or logic outside of the core fine-tuned model, ensure these steps are consistently applied in both environments.
• Model Stability: While your model achieves perfect precision on the test file, it's possible there are edge cases or subtle variations that are not consistently handled, particularly if the model had limited training data. Slight differences in pre-processing (as covered above) can trigger these inconsistent cases.

4. API-Specific Considerations:

• Caching: Some APIs might employ caching to improve performance. If there's a caching issue, you might see outdated results despite making new requests. Try explicitly disabling caching if possible or force a re-processing.
• Load Balancing/Server Differences: If the API service uses a load-balanced architecture, slight variations in server configurations could lead to different processing behavior.
• Rate Limiting: If you are sending many API requests, rate limiting could be affecting results, although this is less likely to impact a specific field extraction.

Troubleshooting Steps:

1. Start Simple: Begin by creating a barebones API call with only the essential parameters and no extras. Gradually add parameters back until you encounter the inconsistency.
2. Isolate the Issue: If possible, try the API call against a different example document (but still similar). This could reveal if the issue is specific to the original document.
3. Reproducibility: Repeatedly run the API call with the same document and settings. If the extracted value is sometimes correct and sometimes not, that indicates an instability or timing issue.
4. Simplify the Model: As a last resort, try using a much simpler, non-fine-tuned model to compare the results against your custom one. This might help you identify if the problem resides in the core logic or in the fine-tuning itself.

You may refer to these documentations for more information:

• Managing processor versions
• Evaluate performance

Here is a similar case that you may find useful as well.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[andressasoares]

Thank you very much for the response! I realized that my function for processing was ignoring the processor version. Once I adjusted that, it worked perfectly.