Troubleshooting Common Chromium Errors

[rdunlavy]

The Chromium renderer is the process on the Looker server responsible for creating PDFs and PNGs of dashboards and Looks. Errors from Chromium or image render jobs in general can tend to be a bit generic. This page provides an overview of the most common Chromium image rendering errors and how to troubleshoot them. 

These errors most commonly occur with customer hosted Instances, as Chromium is automatically installed and updated on Looker hosted instances.  
 

First, Make sure Chromium is installed
 

The first step to troubleshooting any image rendering error is to confirm that Chromium is installed correctly, and that Looker can access it by running the following commands:

chromium --version

This should return something like:

Chromium 70.0.3538.77 Built on Ubuntu , running on Ubuntu 16.04

Next, make sure Chromium is in the proper place, which is /usr/bin/chromium: 

which chromium

The output should be:

/usr/bin/chromium

Running these two commands confirms that Chromium is installed correctly on a supported version (versions > 65, excluding version 71), and that it is in the proper path for Looker to find. Also make sure Looker has permissions to execute chromium.

RendererNoRenderFinishedEventReceivedError

This is a common catch-all error that often appears on scheduled jobs and is associated with the other common errors mentioned below. It is thrown by the Chromium renderer, which - as mentioned above - is the process on the Looker server responsible for creating PDFs and PNGs of dashboards and Looks.

It is essentially a  timeout error - the renderer was waiting for something to finish, such as a SQL query or visualization, but never received confirmation that the task was completed.

Common causes

1. Query cache has expired. Normally, queries are run right before the renderer process so that the results are cached and ready for the renderer to use. However, if the cache expires before the renderer is able to use those results, then you may see this error. This commonly happens when a dashboard has generated a lot of queries or long-running queries. We have seen this resolved by increasing the cache duration on the underlying model or models.
    

2. A custom visualization is missing its completion response. The renderer listens for visualizations to fire a “finished” event when they’re done drawing and ready to be captured. The renderer also polls for a count of visualization elements that are in a “loading” state and will accept the state where that count is 0 as a valid replacement for the finished event. The generic RendererNoRenderFinishedEvenReceivedError error is raised when neither of those conditions happen.
   
   If a custom visualization's code doesn't properly signal its completion, then the renderer may be left waiting. To check if this is the cause, load the dashboard in the browser and check the Javascript console (command+option+j on Chrome) for errors.
    

3. Chromium is not properly installed. On Looker-hosted instances, Chromium is installed by default, but this error has appeared for many reasons on customer-hosted instances. Make sure you've followed the Looker-provided Chromium installation instructions closely.

   1. On Ubuntu, we have seen this resolved by running sudo apt-get install fonts-freefont-otf

   2. For CentOS, we have seen this resolved by running sudo yum -y install liberation-*

 

Render error: Script timed out

When Looker starts Chromium to render a PDF or PNG, it runs rendering-specific JavaScript to check for when all of the content (tiles on a dashboard, or a query for a Look) finishes loading before taking a screenshot of the content. 

After the Chromium process starts running, it has 30 minutes to finish rendering before it will timeout with this error. Since all queries associated with a render job are run and cached before the Chromium process is started, the rendering should complete within the time limit. However, the  Render error: Script timed out error in the UI indicates when the rendering process times out.

If the error message includes the word "retried", this means the renderer (Chromium) tried to execute twice.

Common Causes

1. Query cache has expired. Normally, queries are run before the renderer process so that the results are cached and ready for the renderer to use. However, if the cache expires before the renderer is able to use those results, then you may see this error. This commonly happens when a dashboard has generated a lot of queries or long-running queries. This can potentially be resolved by increasing the cache duration on the underlying model or models.
    

2. A custom visualization is missing its completion response. The renderer listens for visualizations to fire a “finished” event when they’re done drawing and ready to be captured. The renderer also polls for a count of visualization elements that are in “loading” state and will accept the state where that count is 0 as a valid replacement for the finished event. The generic RendererNoRenderFinishedEvenReceivedError error is raised when neither of those conditions are met.
   
   If a custom visualization's code doesn't properly signal its completion, then the renderer may be left waiting. To check if this is the cause, load the dashboard in the browser and check the Javascript console (command+option+j on Chrome) for errors.

3. Unexpected behavior with certain tiles. There are some cases where certain tiles never send the "done" message, even if they have successfully completed. Open a support request with the Department of Customer Love for assistance with confirming whether you are experiencing this unexpected behavior.
    

Could not start Chromium

 

This error most commonly appears on customer hosted Looker instances. 

Common causes 

1. Chromium fonts are not installed. To confirm that this is the case,  the command chromium --headless --disable-gpu --remote-debugging-port=9222  will return a  core dumped error. Visit the Customer Hosted Installation of Rendering Software documentation page for font installation instructions.

2. There is not enough memory or swap space. Confirm that enough memory or swap space has been allocated to the Chromium renderer. 
    

Could not connect to dev tools
 

This error most commonly appears on customer hosted Looker instances. 

This usually means that either Chromium was started and immediately failed, or Chromium started but then could not reach the dev tools in order to control the steps necessary to render content. 

Common causes 

1. Looker is running as the root user. Looker needs to run as a non-root user. When a Chromium process begins, it runs as the same Looker user. Chromium requires being run from a normal account (non-root) to support its security sandbox, and Looker does not have code in place to drop privilege when starting Chromium. To make sure Looker is running as a non-root user:
   
   Running the the following remote debugging port command below:
   `chromium --headless --remote-debugging-port=9222`
   
   If the following output is returned, Looker is running as the root user:
   
   [0308/160359.743951:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported’

If the message below is returned instead, it is more likely that fonts were not installed:  

[0100/000000.982492:ERROR:zygote_linux.cc(621)] Zygote could not fork: process_type  renderer numfds 5 child_pid -1

Visit the Customer Hosted Installation of Rendering Software documentation page for font installation instructions.