So I am trying to use firestore with google's ruby client and running the code sample provided in documentation and I am getting DeadelineExceeded error. Is there any way to diagnose the reason why this is happening?
There doesn't seem to be any logs to examine for firebase. This isn't some crazy 1000qps example either.
Interestingly the system has been working correctly in the past. The service doesn't appear to be offline either.
I would open a ticket with support but I really don't want to pay for Google to fix their issues.
If there are any suggestions on how to diagnose this?
require "google/cloud/firestore"
firestore = Google:☁️:Firestore.new project_id: <my-project-id>
# Get a document reference
doc = firestore.doc "users/list"
data = doc.get
puts data.inspect
/Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-v1-0.10.0/lib/google/cloud/firestore/v1/firestore/client.rb:784:in `rescue in batch_get_documents': 4:4:Deadline Exceeded. d
16 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-v1-0.10.0/lib/google/cloud/firestore/v1/firestore/client.rb:746:in `batch_get_documents'
15 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/service.rb:74:in `get_documents'
14 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/client.rb:348:in `get_all'
13 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `each'
12 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `first'
11 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `get'
10 from foo.rb:7:in `<main>'
9 /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/gapic-common-0.19.1/lib/gapic/grpc/service_stub/rpc_call.rb:127:in `rescue in call': 4:4:Deadline Exceeded. debug_error_string:{UNKNOWN:Deadline Ex
8 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/gapic-common-0.19.1/lib/gapic/grpc/service_stub/rpc_call.rb:121:in `call'
7 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/gapic-common-0.19.1/lib/gapic/grpc/service_stub.rb:156:in `call_rpc'
6 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-v1-0.10.0/lib/google/cloud/firestore/v1/firestore/client.rb:779:in `batch_get_documents'
5 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/service.rb:74:in `get_documents'
4 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/client.rb:348:in `get_all'
3 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `each'
2 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `first'
1 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `get'
18 from foo.rb:7:in `<main>'
1 /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status': 4:Deadline Exceeded. debug_error_string:{UNKNOWN:Deadline
2 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/active_call.rb:186:in `attach_status_results_and_complete_call'
3 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/active_call.rb:175:in `receive_and_check_status'
4 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/active_call.rb:458:in `rescue in server_streamer'
5 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/active_call.rb:457:in `server_streamer'
6 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/client_stub.rb:343:in `block (2 levels) in server_streamer'
7 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
8 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/client_stub.rb:342:in `block in server_streamer'
9 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/gapic-common-0.19.1/lib/gapic/grpc/service_stub/rpc_call.rb:123:in `call'
10 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/gapic-common-0.19.1/lib/gapic/grpc/service_stub.rb:156:in `call_rpc'
11 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-v1-0.10.0/lib/google/cloud/firestore/v1/firestore/client.rb:779:in `batch_get_documents'
12 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/service.rb:74:in `get_documents'
13 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/client.rb:348:in `get_all'
14 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `each'
15 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `first'
16 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `get'
17 from foo.rb:7:in `<main>'
18 /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/active_call.rb:456:in `run_batch': call#run_batch failed somehow (GRPC::Core::CallError)
19 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/active_call.rb:456:in `server_streamer'
20 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/client_stub.rb:343:in `block (2 levels) in server_streamer'
21 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
22 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/grpc-1.55.0-x86_64-darwin/src/ruby/lib/grpc/generic/client_stub.rb:342:in `block in server_streamer'
23 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/gapic-common-0.19.1/lib/gapic/grpc/service_stub/rpc_call.rb:123:in `call'
24 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/gapic-common-0.19.1/lib/gapic/grpc/service_stub.rb:156:in `call_rpc'
25 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-v1-0.10.0/lib/google/cloud/firestore/v1/firestore/client.rb:779:in `batch_get_documents'
26 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/service.rb:74:in `get_documents'
27 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/client.rb:348:in `get_all'
28 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `each'
29 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `first'
30 from /Users/<user>/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/google-cloud-firestore-2.13.0/lib/google/cloud/firestore/document_reference.rb:168:in `get'
31 from foo.rb:7:in `<main>'
The error you're experiencing, DeadlineExceeded
, typically means that the operation you're trying to perform is taking longer than the allowed time, or deadline. In the context of Google Cloud Firestore and the Google Cloud Platform (GCP), this might occur for a variety of reasons:
Network Latency: If the client (in this case, your Ruby application) is far from the GCP region where your Firestore database is hosted, the network latency might be causing your requests to take longer than expected.
Large Documents: If the document you're trying to read is large, it might be taking more time to fetch than the Firestore client's default timeout allows.
Highly Contended Documents: If a lot of operations are trying to read and write to the same document at the same time, it can cause contention and lead to operations taking more time than expected.
To diagnose the issue, you can try the following steps:
Check Network Latency: If possible, try running your Ruby script from a different network or location, or change the GCP region of your Firestore database (please note this might require creating a new Firestore database, as you can't change the region of an existing one).
Check Document Size: If the document you're trying to fetch is large, consider breaking it down into smaller documents or collections.
Check for Contention: If the document you're trying to fetch is being read or written to by many operations at the same time, consider restructuring your data to reduce contention. For example, if you're storing counters in your documents, consider using a distributed counter pattern.
Increase Timeout: If the above steps don't help, you can try increasing the timeout for your Firestore client. Please note that this is generally not recommended, as it might lead to your application hanging for long periods of time. Here's how you can do this in your Ruby script:
require "google/cloud/firestore" firestore = Google::Firestore.new( project_id: "<my-project-id>", timeout: 60 # Timeout in seconds ) doc = firestore.doc "users/list"data = doc.get puts data.inspect
If none of the above suggestions work, it might be worth reaching out to Google's support, even if it's not ideal. Sometimes, issues like these can be caused by factors outside of your control.