How to send images as a response in rag app in dialog flow cx

[JossG]

I have a RAG application built with Dialogflow that retrieves data from a bucket storage containing unstructured data in PDF format. I want to include relevant images in response to the user's question and send them in the output. How can I achieve this?

Is it possible this in a custom payload? The image will vary, of course. Can images be extracted directly from the PDF, or do they need to be pre-extracted and stored separately?

Additionally, if the response does not contain an image, how can I display only plain text without the rest of the custom payload structure?

Ex. User: Show me the diagram fof last month's report. AI Assistan: here's the image: {applicable image}

[ibaui]

Hi @JossG,

Welcome to Google Cloud Community!

To achieve your goal of including relevant images in the response to user queries in your RAG (Retrieval-Augmented Generation) application built with Dialogflow, you can indeed use a custom payload. Here's how you can achieve this, addressing your questions:

1. Image Extraction and Handling:

• Direct Extraction: You can extract images directly from PDFs using libraries like pdfminer.six in Python. This allows you to dynamically retrieve images based on the user's query.
• Pre-Extraction (Recommended): For better performance and scalability, consider pre-extracting images from your PDFs and storing them separately in your bucket storage. This way, you can quickly retrieve images without processing the entire PDF each time.

2. Custom Payload for Images:

• Dialogflow CX's Rich Messages: Dialogflow CX supports rich messages, which allow you to include images, cards, and other interactive elements in your responses.
• Custom Payload Structure: You can define a custom payload structure to include both text and image information.

3. Handling Responses Without Images:

• Conditional Logic: Use conditional logic in your Dialogflow fulfillment code to check if an image is available for the response. If not, send only the text.
• Custom Payload Structure: Maintain the same custom payload structure, but set the image field to null if no image is available.

You can read through this documentation regarding Dialogflow CX Messenger fulfillment.

I hope the above information is helpful.

[avbendre]

How can we extract image from the html