Where I work we’ve set up OpenTelemetry SDK in the applications to expose traces, logs and metrics.
Grafana agent as OTEL collector on the application hosts, Grafana Tempo as backend for traces, Loki for logs and Prometheus for Metrics.
The cool thing about Tempo it generates metrics for ingested spans and their labels (spanmetrics) so this allows us to explore “unknown unknowns” as the author calls it in a very cost efficient way.
Grafana agent as OTEL collector on the application hosts, Grafana Tempo as backend for traces, Loki for logs and Prometheus for Metrics.
The cool thing about Tempo it generates metrics for ingested spans and their labels (spanmetrics) so this allows us to explore “unknown unknowns” as the author calls it in a very cost efficient way.