How to migrate data from Google cloud monitoring to self managed prometheus in GKE cluster?

[Hepzi]

We've been using Google cloud monitoring for one year to collect and monitor metrics for our production workloads. Now, we are planning to migrate all historical monitoring data, alerts, and dashboards from GCM to a self-managed prometheus setup running in our GKE cluster.

I noticed that Stackdriver exporter can be used to pull live data into prometheus, but it doesn't support historical data migration it seems.

What's the way to move historical data from GCM and push it into prometheus? How do I migrate existing alerts and dashboards from GCM to prometheus & grafana?

Does anyone know this? If so, please help me with the solution.

[jayeshmahajan]

1. Export GCM metrics to BigQuery (for historical data)

• Use Cloud Monitoring export to BigQuery to export metrics to BigQuery.
• Query historical time-series data from BigQuery.
• Write a custom script to:
  • Read historical data from BigQuery.
  • Format it in Prometheus' remote write protocol or use OpenMetrics format.

Push to Prometheus via remote write endpoint, or via a bridge like VictoriaMetrics, Thanos, or Mimir.


2. For dashboard and alerts there is no native tools available. I would use gcloud commands and custom script to pull and import those to prometheus and grafana. 

[reinc]

Hi @Hepzi,

Welcome to Google Cloud Community! 

Migrating from Google Cloud Monitoring (GCM) to a self-managed Prometheus setup in GKE, particularly when including historical data, is a complex task but you may check this related documentation about Migrating From GCP Cloud Monitoring to Prometheus and Grafana on Akamai as it have similar to your question.

As an overview in Google Cloud Managed Service for Prometheus. It combines the familiar Prometheus experience with Google's robust and scalable Cloud Monitoring infrastructure, potentially making the migration easier in the long run.

For getting your historical data, you can use Google Cloud Logging. Create 'sinks' to filter and export logs from specific applications. Once filtered, you can download your logs in CSV or JSON file format from the Query result toolbar. 

To ensure a successful migration, consider reaching out to our Google Cloud Support team. Their expertise can help simplify the process and answer any specific questions you may have.


Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.