Scheduling (unlimited) Results to S3

[zachtaylor]

#The Basics

As of Looker 4.2, you can schedule reports directly to an S3 bucket. Results can be unlimited, allowing users to schedule and send large result sets, provided they meet the streaming criteria (that is, the report can’t contain table calculations or totals or, in some dialects, pivots). The scheduler will let you know as you’re scheduling if the report can be unlimited or not. To see this option in the scheduler modal, the user needs to have the send_to_s3 permission.

When to Schedule to S3

Sending reports directly to S3 works well when email is not an option because of the size of the result set. Because we use Sendgrid to process scheduled emails from Looker, and it has a 19.5MB limit (as of November 2016), the reports we send via email have to be limited in size. Streaming to S3 allows customers to bypass browser, memory and email limitations. This also may be useful if you want to automate a system to pull data down from S3 and use it in other applications within your business or in other tools.

Another use case is customers that want to store snapshots or periodic reports in S3.

#How It Works

1. Create a schedule for a saved Look or Dashboard.
2. Select ‘Amazon S3’ from the ‘Destination’ dropdown and enter S3 credentials (the user needs to have the send_to_s3 permission to see this)
3. Select ‘Schedule’ or send a test via ‘Send Test’

Custom Applications

Have a great idea for ways that this feature can be used? Post details below to give ideas to others in the Looker community!