Completing the predictive analytics loop with Looker and Dataiku

[LookerExperts]

This content, written by Looker Alliances Team, was initially posted in Looker Blog on May 13, 2020. The content is subject to limited support.

What data scientists need is a way to streamline the process from data storage, analytics processing, machine learning work, and then back to storage again for further visualization and analysis. It’s also important that data governance and trust in data is maintained throughout this process, as it is essential to meaningful results and insights.

Looker and Dataiku teamed up to streamline this process and empower data scientists with an end-to-end solution for data prep, machine learning, and access to trusted data for rapid reporting and insights.

What does this integration do?

The enables users to seamlessly take prepared and trusted Looker query results and send them directly to Dataiku for machine learning, eliminating manual and cumbersome steps often involved in creating a data science workflow.

With this integration, the processes for completing predictive analytics projects are simplified for tasks like:

• Data prep for analytics
  Dataiku provides cleansing, processing, and enrichment capabilities that return valuable datasets back to Looker users. The resulting datasets can be piped into any of Dataiku’s data warehouse integrations, where data is then able to be queried by Looker.

• Data prep for LookML
  Looker's repository of business logic as defined in helps prepare data for further ML model training and evaluation. Plus, with the Looker query plugin, users can surface trusted datasets curated in Looker to the Dataiku platform. From there, Dataiku’s automated ML features make it possible for any analyst, no matter their level of expertise, to reap the benefits of data science without advanced coding, engineering, or even training.

• Completing the loop
  Users can access machine learning training and evaluation results in Looker for visualization, exploration, and action by writing back results into the source database.

To complete any of the above tasks, users simply tell Dataiku which Looker query to run and use as its training and test dataset - and that’s it! The below graphic shows the step-by-step setup instructions for deploying a model, and the following video highlights the power of this new integration.

How to get started

Here’s how to get started with the Looker query plugin for Dataiku DSS

1. Similar to Looker, Dataiku leverages the performance of data warehouses by offering transformation pipelines running in-database (SQL) or in-cluster (Spark, Hadoop). Use cases include data cleansing at scale, data enrichment, and data processing. Results of these operations are then available for analytics in the database.

2. Looker connects to the database containing results of Dataiku prep, processing and ML as well as other centralized data sources from applications, transactional databases, etc. Business logic is defined in LookML such that all reporting leverages the same definitions.

3. Datasets can be curated in Looker as reports based on these definitions and sent to Dataiku through the Looker query plugin. The plugin leverages Looker’s API to surface the Looks in the Dataiku platform. The plugin additionally requires a dedicated Python environment to be configured.

4. Once the plugin has been set up, a Look is selected in Dataiku and can be synced to a local copy on the platform if desired. As an additional option, cleanup on the dataset can be performed to get the data in the necessary format for machine learning.

5. A machine learning model is run against the dataset provided by Looker.

6. Results of the model (predictions) can be written back to the database and can then be visualized with Looker.

7. A production workflow can be established to continually run the model and gather predictions.

Sample production workflow in Dataiku DSS

Wrapping it all up

This Looker plugin for Dataiku DSS takes a new approach to enterprise-wide data science by not only making it easy to create machine learning models, but also by creating a loop that allows users to further analyze predictive analytics results in their BI layer.

For the full details on how to set up the Looker query plugin in Dataiku, check out the or reach out to our or teams to learn more.