YAML parsing error when transferring to a playbook

[yvesnormandin]

I have the following error when transferring from one playbook to another:

Error! Got action "pb^Flat tire flow" from LLM, but failed to parse its input as YAML:

  preceding_conversation_summary: "Caller, Robert Williams, with phone number 555-555-1212 and zip code 91236, reported a flat tire and is not in immediate danger.",
  caller_name: "Robert Williams",
  caller_address: "789 Pine Lane, Springfield, MA 91236",
  vehicle_description: "2022 Chevrolet Equinox, White"

The error does not occur all the time and it seems to be related to the use of input parameters. If I remove the input parameters, the error no longer occurs.

I am using gemini-1.5-flash-002.

Is there a way to avoid this error?

[Mizar]

@yvesnormandin can you show how you are sending these parameters?

[yvesnormandin]

Sure, I simply defined input parameters:

[MarvinLlamas]

Hi @yvesnormandin,

Welcome to the Google Cloud Community!

It looks like you are facing an issue with the inconsistent and unreliable generation of valid YAML by the ‘gemini-1.5-flash-002’ LLM, especially when input parameters are involved. This unpredictability leads to parsing errors, disrupting workflows and making automation unreliable.

Here are the potential ways that might help with your use case:

• Prompt Engineering: To help you generate valid YAML, you should provide clear instructions, use your delimiters like ‘---’ to separate the YAML block, and include your well-structured example as a template.
• Consider Fine-tuning: If you're generating a lot of YAML, fine-tuning your model can greatly improve its accuracy.
• YAML Linting: You may want to use a YAML linter to automatically check your generated YAML for errors before you try to use it, and you can integrate it into your workflow.
• Error Handling: You may implement robust error handling in your system to effectively catch and correct parsing errors.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[yvesnormandin]

Hi @MarvinLlamas ,

The solution you proposed is not applicable, unfortunately.

I'm afraid I wasn't completely clear in my problem description. The problem I described is in the context of Google Conversational Agents (Dialogflow CX) Playbooks. I was describing a problem that occurs when running an application using playbooks.

Playbooks don't provide any means to do any kind of prompt engineering other than through modifications to the Goal, Instructions, Examples, and Tool OpenAPI Schemas. We can't even see what prompt is being sent to the LLM, nor can we see the LLM completion. This is all hidden inside a black box (a very annoying observability problem). If there is YAML formatting happening, it's all hidden and if there is a YAML formatting problem, we only discover it indirectly, as I did with the fatal error I described in my original post. So this excludes any YAML linting.

Fine-tuning is also excluded since Playbooks don't support fine-tuned models. In fact, they only support a very limited set of Gemini models.

Additional note: There are many cases where the use of Playbook input parameters cause no problem whatsoever, but I haven't been able to figure out why the problem occurs and what to do to avoid it when it occurs, other than removing input parameters.

Any hints would be appreciated. Thanks!