How to convert Cloud trace spans to metrics and explore them in Cloud Monitoring metric explorer ?

[Santosh-Perumal]

Hi Team,

I have my java application running on GKE instrumented with Open Telemetry auto instrumentation and the traces are exported to cloud trace using google exporter-auto-0.28.0-alpha-shaded.jar I am happy with the trace and spans able to see them in cloud trace.

Now I am looking for options to extract metrics out of the spans that I see in cloud trace spans like the latency, response time etc. into useful charts in Cloud monitoring any suggestion on how to get metrics out of cloud trace spans this would really help.

Something that I can explore from metric explorer for cloud trace and then create charts for the metrics that I see in spans is what I am really looking at as I can see I don't see any metrics which GCP pulls by default from cloud trace except the one for BigQuery which I am seeing as I have exported the traces to BigQuery.

When I searched, I see lot of options one of them is to setup OTEL Collector and use spanmetrics connector but not really sure how the collector will be able to store the metrics.

References
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricscon...

[LeAnt]

Hi @Santosh-Perumal 

Welcome to Google Cloud Community!

Cloud Trace spans themselves aren't directly convertible to metrics for viewing in Cloud Monitoring's Metrics Explorer. Metrics Explorer offers a limited view of Cloud Trace data. For detailed analysis, consider using BigQuery directly to explore your trace data. You can leverage Cloud Trace's export functionality to send span data to BigQuery and then analyze it using Metrics Explorer. However, be advised that this product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support.

Steps to follow:

• Configure Cloud Trace Export -- Enable Cloud Trace export to BigQuery. This creates a dataset in BigQuery containing your trace spans.
• Metrics Explorer Query -- In Metrics Explorer, select "Cloud Trace" as the resource and "Bigquery_export" as the metric category.
• Within "Bigquery_export" metrics, choose "Spans Exported to BigQuery" to view the number of exported spans.expand_more.
• You can add filters based on specific timestamps or project information to refine your view.

You won't be directly querying Cloud Trace metrics, but rather the data exported from Cloud Trace spans stored in BigQuery.

I hope this information is helpful.

If you need further assistance, you can always file a case with our support team.

[Santosh-Perumal]

Hi @LeAnt 

Thanks for the response I am already exporting the spans to BigQuery and I am able to see the spans coming up as rows and columns in BigQuery and I am able to query the span data in BigQuery but I would like to further extend this but adding some charts in Cloud monitoring using these metrics.

The metric "Spans Exported to BigQuery" that you mentioned for cloud trace resource gives me just the spans exported counts, but I am basically looking at the other metrics out of my spans like the span.endTime, span.startTime, span.spanId, extendedFields.traceId and other details that I see in the Bigquery.I understand I would be able to query Cloud trace metrics directly but If I can fetch at least exported span details from BigQuery to my Cloud monitoring metric explorer, it will really help me to design a meaning full chart in Cloud monitoring. I understand Google don't fetch this as a metrics by default instead the cloud trace has its own backend where it maintains the data to render it in Cloud trace UI.

Not sure if I need to explore something on the below lines, please let me know if the below options are a feasible one? because currently exporting the trace to bigquery doesn't really helping me to get all the span attributes or metrics from cloud trace in cloud monitoring metric explorer.

Reference  
https://cloud.google.com/trace/docs/otlp?hl=en#otlp-traces

[DocKnow]

It would be really nice if we could use the spanmetrics connectors to ingest into metrics.
It looks like the histogram format is not compatible ...