Trace Explorer With Span Metrics

The Trace Explorer in Sentry is designed to make performance investigation easier and more intuitive. You can now explore span samples, visualize span attributes, and aggregate your data with flexible queries and filters. This guide will walk you through the key concepts and features of the Trace Explorer.

Key Concepts

Traces

• A trace represents a series of transactions or requests through a single system. This includes things like user browser sessions, HTTP requests, DB queries, middleware, caches and more.
• It captures a series of operations (spans) that show how different parts of your application interacted during each transaction.
• Each trace is identified by a Trace ID, which you can use to follow a request across services or projects.
• A distributed trace is made of transactions and spans that traverse multiple servers, clients, edge workers, and other parts of your system. It is assembled by passing the Trace ID through browser headers.

Spans

• A span represents an individual operation within a trace. This could be a database query, HTTP request, or UI rendering task.
• Each span has:
  • Attributes: Key-value pairs like http.method, db.query, span.description, or custom attributes like cart.value, provide additional context that can be useful for debugging and investigating patterns. These are either numbers or strings. Note: numeric span attributes can be used to calculate span metrics, shown below.
  • Duration (span.duration): The time the operation took, used to measure performance.

Span Metrics

• Span metrics are derived from applying a function to your span attributes (default or custom), like p50(cart.value) or sum(ai.token_use), over a granular time frame. This calculation extrapolates metrics that then populate dashboards and alerts. based on that rolling time window.
• These 'metrics' aren't stored separately from your span data. Rather, they're queried on the fly. This reduces your need to manage, ingest, and store custom metrics.

Practical Examples

• See your changes in action: Search for span.domain is localhost:XXXX and sort by Timestamp to see traces from your dev environment.
• Diagnosing slow pages: Search for span.op:navigation and visualize p90(span.duration) to pinpoint slow page loads.
• Finding problematic API calls: Aggregate by http.url and filter where avg.(span.duration) > 2s to identify slow external API calls.
• Database query analysis: Aggregate by db.query and sort by avg(span.duration) to find inefficient queries.

With these tools, the new Trace Explorer gives you powerful ways to understand your application's performance, identify bottlenecks, and make informed optimizations.

Browsing Span and Trace Samples

Trace Explorer allows you to browse span and trace samples and inspect them. Here's how:

1. Search for Samples:

   • Use the search bar to filter results based on default or custom attributes, tags, and strings like http.request_method is "GET
   • Edit and add columns in the span samples table to view any attribute
   • Sort by clicking the column header to toggle between ascending and descending
   • Switch between Span Samples and Trace Samples tabs depending on your focus.

2. Inspect Samples:

   • Clicking on a Span ID or Trace ID opens the Trace Waterfall View, showing the sequence of operations and their durations.
   • You can explore individual spans in detail or see the trace's overall structure.

Visualizing Span Attributes

Trace Explorer makes it easy to visualize metrics aggregated from any span attribute.

1. Set Up Visualization:
   • Use the Visualize section to add metrics like avg(span.duration) or p90(span.duration) to the same chart by clicking Add Series or a different chart by clicking the + icon.
   • You can also create custom visualizations like:
     • P90 of span durations: Useful for identifying outliers.
     • Average cache size: Analyze attributes specific to backend spans.
2. Filter Your Data:
   • Refine your visualization by applying filters in the search bar, such as span.op:db to focus on database spans or http.status_code:500 for errors.

The line chart updates dynamically as you apply filters, helping you track trends and identify anomalies.

Aggregating Span Data

Dive deeper into your data with aggregation capabilities in Trace Explorer.

1. Click into the Aggregates tab.

2. Group By Attributes to investigate patterns.

   • Try http.method to compare GET vs POST performance, db.table to find slow database operations, span.op to investigate patterns in your traces, or custom tags like cart value, CPU or memory usage, or anything else you can think of.

3. Filter and sort your aggregates just like you would for transactions by querying in the search bar. Sort the table of aggregates by clicking the column headers (default: timestamp.)

4. View sample traces that fit your aggregates by clicking the stack icon to the left side of the table rows. Use this to find outliers, debug issues, and find complex performance optimizations.

Compare Attributes (Beta)

When investigating performance issues, you often need to answer: "What's different about this spike compared to normal behavior?". Sentry's compare attributes feature lets you analyze attribute distributions between a selected time range and your baseline data without writing additional queries.

This helps you quickly identify whether a latency spike, error surge, or other anomaly is caused by a specific release, geographic region, browser, API endpoint, or any other attribute you've captured in your spans.

How to Use Attribute Comparison

• On the Trace Explorer page, begin by dragging across a chart's timeline to narrow down to the time range that you want to compare. You will see an option to Compare Attribute Breakdowns.

• When you click on Compare Attribute Breakdowns, the attribute comparison charts below will load with the baseline and time range differences. Purple bars on the left are the time range values, and gray bars on the right are the baseline values. Note: Clicking anywhere else on the page while in compare attributes mode will stop the comparison and reset to baseline.

Reading the Bar Charts

Each chart displays two percentages, like 99.8% | 84.6%:

A large difference in these percentages can indicate that the attribute is more (or less) present during the anomaly.

• Taller bars = higher frequency of that attribute value within its cohort
• If the distributions differ significantly, that attribute may explain the anomaly

Widgets are ordered based on how often the attribute is present in the selection vs the baseline. The most frequently occurring attributes are shown first.

Example: Debugging a Latency Spike

Suppose your p95 duration chart shows a spike from 3s to 12s. To investigate:

1. Drag-select the spike on the p95 chart
2. Click and choose "Compare Attribute Breakdowns" from the menu that appears
3. Review the breakdowns to find attributes with different distributions:

If, for example, you see that 80% of spans in your selection come from a specific release that only accounts for 20% of the baseline, you've likely found the culprit.

Best Practices

• Look for the biggest differences: Attributes where the selection and baseline distributions differ most are your best leads.
• Check coverage percentages: Low coverage (e.g., 4.7% | 0.5%) means the attribute isn't populated for most spans — take findings with caution.
• Use Group By together: Combine attribute comparison with the Group By feature in Visualize to see trends broken down by your suspect attribute over time.

Create Alerts and Dashboard Widgets From Queries or Compare Queries

You can create Alerts and Dashboard widgets from your queries by clicking the "Save As" button:

You can also run side-by-side comparisons of different queries to analyze changes or differences in span data.

Practical Examples

• Diagnosing slow pages:
  • Search for span.op:navigation and visualize p90(span.duration) to pinpoint slow page loads.
• Finding problematic API calls:
  • Group by http.url and filter where avg(span.duration) > 2s to identify slow external API calls.
• Database query analysis: Group by db.query and sort by avg(span.duration) to find inefficient queries.

How Sampling Affects Queries in Trace Explorer

You can use Sentry's SDKs to sample spans and control the volume of data sent to Sentry. Not every span gets captured when sampling is enabled, but Sentry will still give you accurate aggregate results by extrapolating the sampled data.

How It Works

Every span sent to Sentry includes its sampling rate. We also derive the sampling weight, which is the inverse of the rate and indicates the number of original spans that this sample represents.

For example:

• At a rate of 10%, the weight is 10.
• At a rate of 50%, the weight is 2.
• At a rate of 100%, the weight is 1.

All spans you send us after sampling get stored.

When you run queries in Trace Explorer, Sentry uses the sampling weight to compute counts, averages, and percentiles.

Example

Let's say you query for spans where span.op:db and get 2 spans in the results:

Counts

Since each span was sampled with different weights, Sentry extrapolates the count:

10 + 2 = 12 estimated spans

Trace Explorer will display Count: 12.

Sums

Similarly, Sentry incorporates the weight into sum:

10 * 100ms + 2 * 200ms = 1.4s

Trace Explorer will display Sum: 1.4ms.

Averages and Percentiles

Averages and percentiles are weighted. You don't just average the 2 spans -- Sentry accounts for the fact that they represent a larger dataset. For example, the weighted average works like this:

(100ms * 10 + 200ms * 2) / (10 + 2) = 117ms (This is the average duration shown)

Sentry treats this as a representation of the average across all 12 estimated spans. You can trust that metrics like avg() and p90() reflect your services' true health, adjusted for sampling.

Sampling Warnings

If your sampling rate is too low for an accurate extrapolation (for example, keeping less than 5% of an already low number of spans), Sentry may show a low sampling rate warning. This lets you know that extrapolation might be less reliable, especially for more restrictive queries.

Even if you sample your spans, Trace Explorer's extrapolation gives you reasonably accurate answers for counts, averages, and percentiles -- without you needing to do anything extra.

When to Turn Off Extrapolation

While extrapolation is usually helpful, if you are exclusively looking to confirm your sampling usage or look exclusively at sampled data, you can turn off extrapolation.

You can disable extrapolation using the settings icon above charts in tools like Trace Explorer.

How to Tweak Client Side Sampling to Find More Samples

While you can set a uniform sampling rate with tracesSampleRate, the tracesSampler function gives you fine-grained control over sampling decisions. With tracesSampler, you can:

• Apply different sample rates to different transactions: Sample critical endpoints at 100% while sampling less important ones at a lower rate.
• Use contextual information: Make sampling decisions based on transaction name, attributes, or other context like user information or request properties.
• Filter out unwanted transactions: Return 0 for transactions you don't want to capture at all.

For example, you might sample your checkout flow at 100% to catch any issues, while sampling your health check endpoint at only 1% to reduce noise. This targeted approach helps you capture the data that matters most while staying within your volume budget.

To configure tracesSampler for your SDK, see the tracing options in your SDK's configuration documentation.

With these tools, the new Trace Explorer gives you powerful ways to understand your application’s performance, identify bottlenecks, and make informed optimizations.