Dialogflow returns an empty list of conversations

[frangoncan]

Dialogflow returns an empty list of conversations, without responses, however there are conversations in the agent. 

I´ve developed a chatbot with dialogflow in the console and now I need to get the conversations. When I try to get the list of conversations inside the agent using the Python Client Library I don´t get any error but also any data. The iterable seems to be empty. The agent is in the global region, the Log Interactions to Google Cloud option in settings is set to on and I can see conversations in the log.

I have no authentication issue because I can get the list of intents and run other methods without error. I´m using the code in the google documentation with some changes:

from google.cloud import dialogflow_v2

# Create a client
client = dialogflow_v2.ConversationsClient()

# Initialize request argument(s)
parent = "projects/agen-test-xfrc/locations/global" # same result using "projects/agen-test-xfrc"
filter = "(lifecycle_state = OPEN OR lifecycle_state = CLOSED)" # same result without the filter.

request = dialogflow_v2.ListConversationsRequest(
    parent=parent,filter = filter,
)

# Make the request
page_result = client.list_conversations(request=request)

# Handle the response
for response in page_result:
    print(response)

I´ve tried a lot of strategies without positive result:

•  As you can see, I´ve forced the filter to retrieve all the conversations, but the filter has no effect, it isn´t the cause.
• I´ve also tryed with the async client because maybe there were a long running but I get also empty responses. 
• Chat with the agent and re train the agent previosuly to run the code.
• Export - import the agent in another region.
• Altough it´s not necessary being the agent in the global region, I´ve inserted the endpoint for the global region in the Client Options. It doesn´t solve it either.

Anyone could help me? Any suggestion would be appreciated.

Thanks in advance.

@EricSchmidt 

[xavidop]

Hi, you have to first activate "Log interactions to Google Cloud" in your Dialogflow console agent's settings page.

Then just call the API (in this example, with a curl command):

curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json" "https://logging.googleapis.com/v2/entries:list" \
--data "{'resourceNames':['projects/YOUR_AGENT_PROJECT'], \
'filter':'logName=projects/YOUR_AGENT_PROJECT/logs/dialogflow_agent'}"

You could also use a gcloud command like:

gcloud logging read "logName=projects/YOUR_AGENT_PROJECT/logs/dialogflow_agent"

found the solution here: https://stackoverflow.com/questions/57993220/how-do-i-retrieve-conversations-data-from-dialogflow-us...

Make sure that you are seeing the history in stackdriver.

And also check that you have permissions to do that:

Authorization requires the following IAM permission on the specified resource parent:

• dialogflow.conversations.list

Best,

Xavi 

[frangoncan]

Hi @xavidop ,

Very appreciated your quick response.

Previously I had already activated "Log interactions to Google Cloud" . I´m pretty sure I´m watching the history in stackdriver, I check firstly the history dialogflow console and then in the log, in the Log Explorer in Observavility Logging (previously Stackdriver). And I have the dialogflow.conversations.list permission.

Both of your code proposals, the API call with curl and the read gcloud command works making them some changes, but they retrive me the complete log,

Why the conversacion list method retrive me empty in the code I sent you? My preference would be to get the conversion list and the conversation messages for each conversation.

Thanks a lot for your help and support.

[xavidop]

In that case what I would suggest is:

1. Check the credentials you have set are properly set

2. Check if running the client using the gRPC and REST protocols work

3. If it returns a long run operation, check if you handling it properly

Check this post for auth reference: https://stackoverflow.com/questions/64054595/how-to-use-dialogflow-client-in-python

I will also recommend to use the latest version of the API:

from google.cloud import dialogflow_v2beta1

[frangoncan]

1. I think the credentials must be properly set because when I request the intents of the agents using the python library I get it, I don´t get any error. I  have set the Application Default Credentials (ADC).

2. Please, could you explain it to me in more detail?

3. I thought about long-running time ago, but I managed it with the below code with the same result: empty response. ¿could you check this snippet, please?

from google.cloud import dialogflow_v2
import asyncio

async def sample_list_conversations():
    # Create a client
    client = dialogflow_v2.ConversationsAsyncClient()

    # Initialize request argument(s)
    filter = "(lifecycle_state = OPEN OR lifecycle_state = CLOSED)"

    request = dialogflow_v2.ListConversationsRequest(
        parent="projects/agen-test-xfrc",filter = filter,
    )

    # Make the request
    page_result = await client.list_conversations(request=request)

    # Handle the response
    async for response in page_result:
        print(response)

await sample_list_conversations()

Regarding the link you sent me that it´s about authentication, I don´t have an authentication issue because there are clients and methods that works, as 

intents_client.list_intents and other.

I have also used the latest version of the API with the same result. 

Thanks a lot for your support, please, if you have any other suggestion don´t hesitate to tell it to me. I´m a bit frustrated with this issue.

[xavidop]

For number 2. The Dailogflow libs sometimes have a REST client and a gRPC client or there is a property that you can use to set which transport you want to use: https://cloud.google.com/python/docs/reference/dialogflow/latest/google.cloud.dialogflow_v2beta1.ser...

I had issues sometimes with gRPC and had to use REST and viceversa

Also I would highly recommend to test without the filter and when you get results, start with the filter.

Another recommendation is to use always v2beta1 version of the APIs.

I think the get_conversation it just retrieve a conversation. You will need to list all the conversations (list_conversations method): 

https://cloud.google.com/python/docs/reference/dialogflow/latest/google.cloud.dialogflow_v2beta1.ser... get the id of every single conversation and call get_conversation.

I think the id of get_conversation is a single id:

[frangoncan]

Firstly, thanks a lot for keeping your support.

My strategy is the same you are telling. In a first step get the list of conversations (list_conversations method) and the get each conversation with get_conversation.

Here you have the code to get the conversation list forcing the transport property,without filter and using the v2beta1 API and I keep getting and empty response:

from google.cloud import dialogflow_v2beta1

# Create a client
client = dialogflow_v2beta1.ConversationsClient(transport='grpc')

# Initialize request argument(s)
parent = "projects/agen-test-xfrc"
request = dialogflow_v2beta1.ListConversationsRequest(
    parent=parent,
)

# Make the request
page_result = client.list_conversations(request=request) #transport ="rest"

# Handle the response
i=0
print(i)
for response in page_result:
    print(response,i)
    i = i+1
print("Finished")

Output:

0
Finished

And finally, I also think the id of get_conversation is a single id but cannot be the one you are showing. You can check that is different in each interaction of the conversacion. Any way, I have checked with the one of those ids and i get:

NotFound: 404 Conversation [308d2bf1] not found in project [agen-test-xfrc].
com.google.apps.framework.request.StatusException: <eye3 title='NOT_FOUND'/> generic::NOT_FOUND: Conversation [308d2bf1] not found in project [agen-test-xfrc]. [resource_name: "projects/agen-test-xfrc/conversations/308d2bf1"
]

Any more idea? I´m getting crazy 😁

Very thanks!

[xavidop]

thanks for the update!!

Can you please make a for loop that gets all the conversations with the list_conversation method and then calls the get_conversation method with the conversation_id set as the name attribute: https://cloud.google.com/python/docs/reference/dialogflow/latest/google.cloud.dialogflow_v2beta1.typ...

[frangoncan]

I understand your approach and It´s my intent. But the issue is that I don´t get any conversation in the list when I use  list_conversations.

Please, could you test the code I sent you in my previous reply?

Thanks in advance.

[xavidop]

sorry, I see what you mean.

at this point, what I would suggest is to use the CloudLogging API to retrieve that info...

I dont have more ideas 😞

Best,

Xavi

[xavidop]

I am getting the same error: empty list

Can you try this?

from google.cloud import dialogflow_v2beta1
from google.oauth2 import *
import json


# Create a client
client = dialogflow_v2beta1.ConversationsClient()
creds_file = "credentials.json"

dialogflow_key = json.load(open(creds_file))
credentials = (service_account.Credentials.from_service_account_info(dialogflow_key))


# Initialize request argument(s)
parent = "projects/test-cx-346408"
request = dialogflow_v2beta1.ListConversationsRequest(
    parent=parent,
)

# Make the request
page_result = client.list_conversations(request=request) #transport ="rest"

# Handle the response
i=0
print(i)
for response in page_result:
    print(response,i)
    i = i+1
print("Finished")

[frangoncan]

Empty list....😖 thanks anyway. 

Regarding your code, how can I get credential.json? My  credentials are set by Application Default Credentials (ADC).

[xavidop]

credentials.json is an API key generated from a IAM Service Account

[frangoncan]

Same....empty list.

from google.cloud import dialogflow_v2beta1
from google.oauth2 import *
import json


# Create a client
client = dialogflow_v2beta1.ConversationsClient()
creds_file = "agen-test-xfrc-08de245f8dd5.json"

dialogflow_key = json.load(open(creds_file))
credentials = (service_account.Credentials.from_service_account_info(dialogflow_key))


# Initialize request argument(s)
parent = "projects/agen-test-xfrc"
request = dialogflow_v2beta1.ListConversationsRequest(
    parent=parent,
)

# Make the request
page_result = client.list_conversations(request=request) #transport ="rest"

# Handle the response
i=0
print(i)
for response in page_result:
    print(response,i)
    i = i+1
print("Finished")

0
Finished

Very appreciated for trying to help me, Xavi. 

[xavidop]

ouch! this is so annoying!

I will suggest to:

1. Create an issue: https://cloud.google.com/support/docs/issue-trackers

2. Use the CloudLogging API

Best,

Xavi

[frangoncan]

I´ll follow your advices after our discussion. In fact, I´m already working on CloudLoging API but the flow will be more complex, it´s a pitty existing  methods as conversiont_list and get_conversation.

Anyway, I´ll open an issue, if I get a solution I´ll post it here.

Very appreciated your support. No words!

Fran

[xavidop]

Nice!
Send me the link of the issue do I can follow up!

Best
Xavi

[rauuuuuuuuuuul]

Hi Xavi 

Hope you are well 

I am exploring a very similar scenario in which I need to return information from DFCX. I cant seem to get the conversation out - I think its the same error

Has there been any update to this / have you developed a solution?

Thanks