Using GCS get within a Workflow returns 404

[mr_cech]

Using a Google Workflow and armed with information from the Google Storage Connector, I had a goal of retrieving the application/json file that was stored via YAML. However, when I try to use the seemingly simple task I am disappointingly met with a 404.

Here is my YAML source on the Google Workflow:

main:
  steps:
  - list_items:
    call: googleapis.storage.v1.objects.list
    args:
      bucket: "processed-items"
    result: item_list
  - get_single_item:
    call: googleapis.storage.v1.objects.get
    args:
      bucket: "processed-items"
      object: "item_2024-01-24.json"
    result: single_item_json
  - returnOutput:
    return: '${single_item_json.entities}'

I have verified that the json object is in the bucket, both visually and because the result of the 'list_items' step shows it in the response body.

Am I using the get method incorrectly?

[alexmoore · Accepted solution]

So I just tried copy pasting your code into a sample project with a file of that name in a test bucket and the `get_single_item` step works as expected.  Is the object definitely there? does the service account of the workflow have access to it?

The last step however might be an issue.  By default the `get` call will return the metadata of the object and there is no value 'entities', so this will fail.  To see what values there are see:

https://cloud.google.com/workflows/docs/reference/googleapis/storage/v1/Overview#object

If however you want to get the json itself, you will need to add the following `alt` parameter to the 'get' call:

  - get_single_item:
    call: googleapis.storage.v1.objects.get
    args:
      bucket: "processed-items"
      object: "item_2024-01-24.json"
      alt: "media"

As per: https://cloud.google.com/workflows/docs/samples/workflows-connector-storage?hl=en

Hope that helps,

Alex

[mr_cech]

Hello Alex,

Thank you for your response. I did confirm that the service account does have access to both the file/bucket/project resources. I did try adding the alt: "json" since I knew that the data was a JSON object. googleapis/storage/v1/objects/get #arguments  But this also didn't work. Only alt: "media".  Once I added the alt: "media" it resolved the 404 error. However, I am now dealing with a response size constraint.

HTTP response exceeded the limit of ##### bytes

Now, my research is going to shift to increasing the response limit.

[alexmoore]

You might be hitting this constraint of 2MB as per: https://cloud.google.com/workflows/quotas#resource_limits

How big is the JSON?

[mr_cech]

I'm pretty confident that the entire contents is larger than 2MB.  However, just the entities portion of it is just a fraction of it. Which is why I was trying to just return it with 

return: '${single_item_json.entities}'

I am now looking into using a Cloud Function to get around the Workflows quota.