Re: Agent not returning toolCall in response

jin280 · 04-19-2024 09:02 AM

I am trying to use Function Tools in Agent Builder. According to the Vertex AI documentation here, I should get back toolCall in the response when the agent thinks that I am calling a tool.

{
  "queryResult": {
    "text": "what is the weather in Mountain View",
    "languageCode": "en",
    "responseMessages": [
      {
        "source": "VIRTUAL_AGENT",
        "toolCall": {
          "tool": "<tool-resource-name>",
          "action": "get-weather-tool",
          "inputParameters": {
            "location": "Mountain View"
          }
        }
      }
    ]
  }
}

I have tried this the following way:

In agent builder playbook, I type in how is the weather. I can see in the network request from dialogflow.clients6.google.com/v3alpha1/ that it contains toolCall and tells me which function is called.

The response with playbook:

  "queryResult": {
    "text": "how is the weather in Stockholm?",
    "languageCode": "en",
    "responseMessages": [
      {
        "source": "VIRTUAL_AGENT",
        "toolCall": {
          "tool": "projects/geoconxaiservices/locations/eu/agents/2e495876-b080-4755-9f98-9962ab0cc873/tools/a11594cb-50f5-42cf-9eba-06728ee91e92",
          "action": "getWeather",
          "inputParameters": {
            "location": "Stockholm, Sweden",
            "unit": "celsius"
          }
        }
      }
    ],

When I use the integration with Dialogflow Messenger, it calls dialogflow.googleapis.com/v3/ and I don't get the toolCall in the response.

  "queryResult": {
    "text": "how is the weather in Stockholm?",
    "languageCode": "en",
    "responseMessages": [
      {}
    ],

But it will also fail on following queryInputs since it is waiting for the output from my client side code execution. I get "message": "com.google.apps.framework.request.BadRequestException: Session is waiting for tool call result of tool, <uuid> and action, getWeather.". I have tested the same flow in both playbook and df messenger (with the same agent) and it doesn't seem that they work the same way. Does anyone know why there is a difference in behavior and why isn't toolCall returned to df messenger?

Nomie

Also having questions about this, I know since it's preview they probably aren't answering these, but just as they made videos explaining how to use the other types of tools (OpenAPI & Data store) it would be nice to have a video explaining how to use the Function Tools 🙂
Videos I'm talking about:

jin280

Yeah I think so too. I guess we'll just have to wait until it's not preview anymore. 😀

Poala_Tenorio

It seems like you're encountering a discrepancy in behavior between using Playbook and Dialogflow Messenger with Function Tools in Agent Builder. The behavior you described suggests that while Playbook correctly returns the toolCall in the response, Dialogflow Messenger does not.

This could be due to several reasons:

Feature Availability: It's possible that the Function Tools feature is not fully supported or enabled for Dialogflow Messenger at the moment. Google may be rolling out updates or improvements to the service, and Function Tools might not be fully integrated into Dialogflow Messenger yet.
Configuration Differences: There might be differences in how Playbook and Dialogflow Messenger are configured to interact with your agent. This could lead to variations in the responses they receive, including whether or not the toolCall is included.
Bug or Issue: There could be a bug or issue causing Dialogflow Messenger not to return the toolCall in the response. This could be on the Dialogflow side or related to how the integration is set up.

To troubleshoot this issue:

Check the latest updates or announcements from Google regarding Function Tools and Dialogflow Messenger integration.
In the meantime, you might need to rely on Playbook for testing and development purposes until Function Tools are fully supported in Dialogflow Messenger, or until you find a resolution to the issue.

quadking

Can we get some more help here? I am observing the same behaving in which the toolCall is no present in the Java DetectIntentResponse(0.59.0). However I have enabled Conversational Logging and I do see it in the Log Explorer:

...
responseMessages: [
0: {
source: "VIRTUAL_AGENT"
toolCall: {
action: "fx-weather"
inputParameters: {
location: "West Chester, PA, 19425"
unit: "fahrenheit"
}
tool: "projects/slamway/locations/us-east1/agents/a5511e33-6877-41e1-824e-4a209dfa4bb1/tools/85f8a7e8-4a47-4a96-bcab-6ca87ad83b50"
}}]
text: "What is the weather in West Chester PA, 19425?"
}
responseId: "d6e9ca94-007c-4536-be40-60d4870b5bd5"
...

It would be nice to know if this is a setup issue, unsupported feature, or a bug...

quadking

It works() with the beta lib: import com.google.cloud.dialogflow.cx.v3beta1.*;