Conversational agents exceeding token limits?

[anwhite]

Good day all,

I'm using conversational agents with a playbook, tied to a third party CCaaS platform.

I've set the input token limit in the agent settings to 8k, and I've set the context token limits to preserve only 4.0k tokens. 

I've noticed, in some scenarios, the logs show errors that indicate that dialogflow is receiving inputs that exceed the token limits (only ever by <150 or so tokens), which triggers DF to drop the session back to the CCaaS, sometimes ungracefully.

As a workaround, I imagine it's possible to restructure the playbooks to never engage in conversations where the turns and instructions could take up so many tokens, but it seems like that would require a number of handoffs and significant complexity to do so for some use cases.

Has anyone else encountered this situation?

Edit: I'm seeing this issue in the tracker https://issuetracker.google.com/issues/355413976

It looks like the context token limits were introduced to combat this, but it's possible it's not implemented in a way that prevents the silent failures.

[marckevin]

Hi @anwhite,

Welcome to Google Cloud Community!

As also stated in the existing feature request, you're likely encountering prompt bloating, where previous conversation turns are included in the request to maintain context as the conversation progresses, which results in exceeding the token limit. As a direct mitigation, you can optimize and shorten your playbook instructions and examples to reduce the conversation history and help avoid hitting the token limit.

I noticed that there is already feedback on the bug, which utilizes the context token limit to shorten the conversation history and still hitting the token limit. If you haven't upvoted it yet, consider upvoting the request, as shown in the image below.

Note that there’s no definite date as to when this feature request will be implemented. For future updates, I suggest keeping an eye out on the feature request and the Dialogflow release notes.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.