BlockingIOError [Errno 11] write could not complete without blocking on cloud run services

[surshrestha]

We have lot of  services(python) on cloud run, recently without any change in revision, many of the apis are failing with this error. BlockingIOError [Errno 11] write could not complete without blocking


with RCA, we were able to pin point that all of these are at logger. Mostly on the places where we're logging large dictionaries but this is not the case always. 

I know this is because of some changes cloud run because we have many services(copy of cloud run svc) being deployed on other environments k8s, vms and with the same code its not failing there at all. 

please help

[leomconti]

Hey, just had this issue. I removed the huge payloads I had from the log and it solved my problem. I think in my case it was because I was logging big strings of base64 images in a dict. Weird because I never got those errors before. I found the specific error at the end of the large payload that I was trying to log, by looking at the cloud log on cloud run: `Unable to print the message and arguments - possible formatting error.` Maybe weird strings on the payload? Or too large?

In the previous iterations of my api the logs were working ok.My service (python) was doing `log.info("Received ...", data=data: ExpectedType)`. Where ExpectedType was a python dataclass with no stringify or anything, just plain adding the object to the log. Not sure if this could've caused the issue either, didn't test much. As the log was not essential I just removed the payload.

Hope this helps, if you find any other information, let us know on the thread here.

[florian_d]

I've experienced the same issue with "BlockingIOError [Errno 11] write could not complete without blocking".

I had prints inside "async" functions. It happens with large prints, and sometimes with light ones. I've never had such errors before...

Can somebody from Google gives us some explanations? At the end, our customers are not happy. I've removed all my "prints". So now I'm blind.

If it happens with the "logger" lib as well, we don't have many options.

The unpredictability makes it hard to deal with.