Guidance on Fine-Tuning Llama 3.1 for chatbot on Vertex AI with Unstructured Data (PDF)

[rhirani]

I’m currently working on a project where I need to fine-tune the LLaMA 3.1 model using Google Cloud’s Vertex AI. However, my training data is in unstructured formats like PDFs, and my goal is to create a chatbot for this custom data. I'm exploring two potential methods but facing some issues:

1. Agent Builder: This approach allows me to upload files to Cloud Storage and fine-tune through Agent Builder, possibly integrating with Dialogflow for chatbot creation.

   • Issue: I cannot find an option to select the LLaMA 3.1 model. The available options are limited to default models.

2. Vertex AI Model Garden: Using the notebook interface, I can fine-tune models, but the approach requires data in structured formats like JSON. I do not have a data in conversation form.

   • Issue: My data is in PDF format, so I would need to preprocess it into JSON or a similar structured format.

My Questions:

• Which of these approaches would be best suited for fine-tuning LLaMA 3.1 with unstructured PDF data to create a chatbot?
• Are there any recommended workflows or tools within Vertex AI for converting and processing unstructured data like PDFs into a suitable format for model training?
• Any specific resources or documentation that would help streamline this process?
• If possible, could you also suggest the steps I should follow to accomplish this?

I’d greatly appreciate any guidance, suggestions, or relevant documentation to help me effectively use these methods for my project.

Thank you!

[ruthseki]

Hi @rhirani,

Welcome to Google Cloud Community!

Here’s the answer to your questions:

• Which of these approaches would be best suited for fine-tuning LLaMA 3.1 with unstructured PDF data to create a chatbot? 

LLaMA 3.1 model is not included in Vertex AI Agent option in Vertex AI Agent Builder. This approach might not be suitable unless Google Cloud updates its model options. Below image indicates the available model to choose from:

Additionally, you are right that in Vertex AI Model Garden, it requires structured data formats like JSON. This approach is more flexible for fine-tuning LLaMA 3.1, provided you preprocess your PDF data.

• Are there any recommended workflows or tools within Vertex AI for converting and processing unstructured data like PDFs into a suitable format for model training?

For PDF Data Preprocessing:

Extraction: Begin by extracting text from your PDFs. 

Chunking & Structuring:

Chunking: Divide extracted text into smaller, contextually relevant chunks. This could be paragraphs, sections, or even sentences depending on the document structure and desired granularity.

Structuring: Convert each chunk into a format suitable for conversational training. A common approach is the "Question-Answer" format.

• Any specific resources or documentation that would help streamline this process?

Here’s some documentations that you may check: 

• Parse and chunk documents
• Llama 3.1 documentation
• For more technical information about generation parameters and recipes for how to use Llama 3.1 in applications, please go here.
• If possible, could you also suggest the steps I should follow to accomplish this?

          Fine-tuning LLaMA 3.1 with Vertex AI Model Garden:

• Prepare your data: Upload your structured JSON data to a Google Cloud Storage bucket.
• Choose the appropriate notebook: Select a Vertex AI Model Garden notebook tailored for text generation or question answering tasks.
• Modify the notebook:
• Data loading: Update the code to load your structured JSON data from the Cloud Storage bucket.
• Model selection: Ensure you're working with the LLaMA 3.1 model. You might need to download and integrate the model weights separately as it's not directly available in Model Garden yet.
• Training parameters: Fine-tune the training hyperparameters for optimal chatbot performance.
• Execute the training: Train your LLaMA 3.1 model on your prepared PDF data.
• Evaluate and iterate: Assess the chatbot's performance and make adjustments to your data preprocessing, model parameters, or training process as needed.

           Deployment & Integration:

• Deploy your fine-tuned model: Once satisfied with the performance, deploy your model to Vertex AI for online predictions.
• Integrate with Dialogflow (optional): For a more robust chatbot experience, consider integrating your fine-tuned model with Dialogflow. This allows for natural language understanding, intent recognition, and dialogue management capabilities.

I hope the above information is helpful.

[rhirani]

I am using notebook that opens from model garden while enabling Llama 3.1

• Llama 3.1 Notebook 

So, it fine if I fine tune here using my dataset ?