Configure dataset s3 storage in Document AI from terraform

[niteshag123]

Hi Team,

Can you please help me or suggest me How we can create a dataset storage location(bucket) in document AI GCP through terraform?

[McMaco]

Hi niteshag123,

Welcome to Google Cloud Community!

It's important to understand that Terraform currently cannot directly manage Document AI to create a dataset storage location (bucket). However, you can achieve the desired outcome by combining Terraform for Cloud Storage bucket creation and manual configuration within Document AI. Here's how:

1. Create a Cloud Storage Bucket with Terraform:
   Follow the steps here in using Terraform to create storage buckets and upload objects to create a dedicated bucket for your Document AI documents using Terraform. Remember to choose a unique name, set the appropriate region, and configure access control if needed.
   
   
2. Update Document AI Dataset Storage Location (Manually):
   Once your Terraform script successfully creates the bucket, you'll need to manually update the storage location within your Document AI dataset. There are two main ways to achieve this:
   
   1. Using GCP Console:
      1. Navigate to the Document AI section of the GCP Console.
      2. Select the relevant processor where your dataset resides.
      3. Click on the desired dataset.
      4. In the dataset details page, locate the storage configuration section.
      5. You'll likely see a predefined location managed by Document AI.
      6. Click "Change" and choose "User-managed Cloud Storage."
      7. Provide the path to your newly created bucket within the "Cloud Storage location" field. The format should be gs://your-bucket-name/path/to/folder (replace with your actual bucket name and desired folder structure).
      8. Save your changes
      
   2. Using Document AI API:
      1. If you prefer programmatic management, consider using the Document AI API to update the dataset storage location.
      2. Follow the official documentation on modifying datasets and refer specifically to the gcsSource field for setting the Cloud Storage location: https://cloud.google.com/document-ai

   Key Points:

   • Remember, Terraform creates the bucket, and you need to point the Document AI dataset to it manually through the console or API.
   • Ensure your Document AI service account (or the user making the changes) has the necessary permissions to access the Cloud Storage bucket.
   • Consider organizing your documents within folders in the bucket for better management within your Document AI dataset.

   By combining Terraform's automation for bucket creation and manual configuration in Document AI, you can achieve the desired workflow of storing documents in a dedicated Cloud Storage location for your Document AI processing needs.
   
   I hope the above information is helpful.