The platform for data

[LookerExperts]

This content, written by Frank Bien, was initially posted in Looker Blog on Oct 10, 2018. The content is subject to limited support.

For decades organizations have been using data to better understand trends or events that are happening in and around their business. Today, business systems are generating far more actionable data than ever before – data that becomes even more valuable when intelligently integrated together. But the explosion of SaaS applications has made things far more complex. Organizations that previously had under 100 applications often have 10x that number today. Having a SaaS solution for every problem is great, but each of those solutions brings more data.

Unfortunately, the analytic data toolchain hasn’t kept up. It’s broken. The mess of point “self-service” tools that were designed to operate on narrow sets of siloed data have been cobbled together to create Frankenstacks – technology science projects that are painful to operate and nearly impossible to maintain. And every new data project often means recreating those complicated stacks from scratch. The Business Intelligence “self service, but only for a single piece of the puzzle” has failed us.

It’s time to rethink how data projects get done. On one hand, we have a wealth of valuable business data being generated by every application and system in our companies. On the other hand, we have no common way to create value out of that data. We’re constantly starting from scratch. What’s required is a common surface for that data that can be quickly shaped into more specific data applications that meet our core data needs. For example, the Revenue team needs a data application to help them understand price optimization. The Marketing team needs a data application to help them understand attribution and focus on ad spend. The IT team needs data applications to help them understand the myriad of event data that their systems generate.

Creating a common surface on which to empower these organizations is the key. To modernize the data stack – and greatly simplify the data supply chain to build value out of data in our organizations more quickly – a new platform for data is required.

The Platform for Data.

The idea of a platform is proven – build the underlying infrastructure pieces and make it extensible with application development components to allow the creation of "end applications" to solve business problems more quickly. It’s time to bring this idea to the data world.

Our industry has historically moved away from platforms in favor of one-size-fits-all BI dashboard tools. We now have a mess of point tools that make it harder to solve end-user business problems. Generic applications for data problems are not nearly as valuable as a common surface for data that can be formed and melded together to solve dozens of problems and is more closely aligned to how different business teams work.

The explosion in data volume and complexity has made a Platform for Data even more critical.

The more data you have, the more valuable it gets if it can be integrated across all business operations. Now that the movement of data to the cloud is the norm rather than the exception, we have untold amounts of data sitting in warehouses and lakes waiting to be piped through our daily workflows. BI grew up around the idea of small data extractions, but now we have new fast databases that give us the opportunity to solve much bigger, broader problems.

We’re also seeing the modern workforce hungry for data in areas traditionally ignored by point tools. The movement away from generic dashboards is well underway. In the past sales data was for the Sales team, systems data was for the Operations team, and ad data was for the Marketing team. Now, we see that bringing the sales data together with marketing data has huge value... and that's just the start. Companies want a more complete and integrated modern solution that goes beyond BI and includes specific applications like Customer Success, Marketing Attribution and Event Monitoring. It’s a way to work in data and operationalize your business around accurate, current data.

So, Looker started to build this platform. We’re organizing it around three big ideas:

Core services: If you want to retain flexibility, you have to assume you’ll need to be looking at all of the raw source data, not just subsets for a specific application. Then you need to bring together and rationalize the hodge podge of data products that are needed to make a solution: data preparation, data integration, embellishment, governance, security, caching, visualization, access, etc…. Take what you currently need five different products to do for each task and build it into a single layer.

Extensibility: Integration is the key – how do X and Y come together to mean Z? Take those core services and put them together in an open, web-native architecture. Then build development environments to that integration, develop a language, and make every part extensible and accessible to other systems. The goal needs to be 110% API coverage – meaning all and more of the core service functionality must exist through these APIs.

Applications: In the world of SaaS you can't just build the platform, you have to build the first big applications. For Looker our first big application was BI, but that quickly evolved to address more specific functions, like Marketing, Event Analytics or Customer Success. To achieve true adoption of the platform, we also knew we had to embrace the people who will build third party applications on the platform and give them tools they want to use to deliver business value to their end users. In the world of SaaS, time-to-value is everything, so providing the applications on top is critical.

Our vision is simple: to build a platform for data that easily integrates all of a business's data and then allows it to be melded to specific work processes in a way where users can do more with it and solve higher-value problems. It gives today's workforce what they want: the ability to work in data rather than just viewing it. At today we introduced – the next evolution of the Platform for Data. When you build a platform, you don't even know what people are going to build on it later. Five years from now, we’ll be surprised at what people will create on the platform. Today is the beginning of boundless possibilities.