I have a GKE cluster with ASM. The cloud logging and trace are enabled, the deployments are able to register the sdout output in the logging, but in the cloud trace (in the metrics within anthos) it does not register anything request. The topology trace works, it shows the services communicating, but it doesn't register anything in the cloud trace. What could be missing?
Hi @wagnerww
I can list the key points to consider when addressing the issue of accessing Cloud Trace within Anthos Service Mesh. Feel free to review and provide additional details about your deployments and workflow if you continue to encounter issues.
If you're only observing topology traces, it suggests potential issues with trace sampling or how request propagation is managed.
To correlate trace spans, trace context must be propagated between services. Ensure your applications correctly transmit necessary trace context headers such as x-b3-traceid, x-b3-spanid, x-b3-sampled, traceparent, or x-cloud-trace-context. These headers enable sidecar proxies to associate spans within a single trace. Trace context headers in the supported formats outlined in the provided document (B3, W3C TraceContext, Google Cloud Trace, gRPC TraceBin).
Traces are generated when actual network interactions occur between services. Confirm that your services are genuinely communicating over the network and not bypassing the service mesh in any way.
As per the documentation, tracer configuration is a part of proxy bootstrap configuration. Restart pods to apply updates. If tracer configuration changes have been made, ensure that pod restarts are performed for the changes to take effect.
Verify that sidecar proxies (Envoy) are properly configured and not encountering any hindrances that might prevent trace data transmission.
For further assistance, you can refer to the official documentation on Cloud Trace and Anthos Service Mesh: https://cloud.google.com/service-mesh/docs/observability/accessing-traces. If the issue persists, providing additional details about your setup would greatly aid in pinpointing the root cause.
Hello @devenes ,
This documentation you mentioned, I had already seen. However, it doesn't work:
cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
mesh: |-
defaultConfig:
tracing:
stackdriver: {}
kind: ConfigMap
metadata:
name: istio-rapid
namespace: istio-system
EOF
Error from server (BadRequest): error when creating "STDIN": ConfigMap in version "v1" cannot be handled as a ConfigMap: json: cannot unmarshal object into Go struct field ConfigMap.data of type string