Go setup

Sherlock has no Go distribution and needs none. The upstream OpenTelemetry Go SDK sends what Sherlock expects once four things are set: delta metrics, exemplars on every request, a trace_flags attribute on each exemplar, and Sherlock’s histogram boundaries. One file, sherlock.go, takes care of them. This page adds it to a net/http service. The file comes in two variants that send the same data. In the first, every Sherlock setting is written in the file, so a reader sees all of them in one place. In the second, the standard OTEL_* environment variables carry the settings, which suits a team that already configures OpenTelemetry that way, and the file is shorter. Step 2 has both. Configuration covers production settings such as health checks, custom histograms, sampling, and logs. Differences from Node.js lists what the Go setup does differently from the Sherlock SDK for Node.js.

What you’ll learn

Which modules to add, at which versions
The two places the Sherlock settings can live, and what stays in code either way
What sherlock.go sets, and why each setting is there
Where the first traces and metrics show up

Prerequisites

Go 1.22 or later, for ServeMux method-and-path patterns. Tested with Go 1.25.
A Sherlock organization and its bearer token from Settings → Collector. Get started shows where.
A service on net/http. For other routers, see Routers and nested muxes.

Steps

Add the modules

go get go.opentelemetry.io/otel@v1.46.0 \
  go.opentelemetry.io/otel/sdk@v1.46.0 \
  go.opentelemetry.io/otel/sdk/metric@v1.46.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.46.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp@v1.46.0 \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.71.0 \
  go.opentelemetry.io/contrib/instrumentation/runtime@v0.71.0

These are the versions this page was tested with. Recent contrib releases changed behavior that this page relies on: otelhttp.WithRouteTag was removed and the OTEL_SEMCONV_STABILITY_OPT_IN switch is gone. Keep the pins until you have tested an upgrade.

Add sherlock.go and set the environment

Pick one variant. Save the file next to your main.go and export the variables it expects. Nothing in either file is specific to your routes.

Settings in the file
Settings in environment variables

sherlock.go

// sherlock.go: OpenTelemetry setup for a Go service that sends to Sherlock.
package main

import (
	"context"
	"errors"
	"os"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/runtime"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/metric/exemplar"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

// Setup starts the traces and metrics pipelines. Call it once at start-up.
// Call the function it returns after your HTTP server has stopped; it
// flushes what is still buffered.
func Setup(ctx context.Context) (func(context.Context) error, error) {
	// Both values are on the Sherlock Settings → Collector page.
	endpoint := os.Getenv("SHERLOCK_ENDPOINT")
	token := os.Getenv("SHERLOCK_ACCESS_TOKEN")
	if endpoint == "" || token == "" {
		return nil, errors.New("set SHERLOCK_ENDPOINT and SHERLOCK_ACCESS_TOKEN")
	}
	headers := map[string]string{"Authorization": "Bearer " + token}

	// service.name and env come from OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES.
	res, err := resource.New(ctx, resource.WithTelemetrySDK(), resource.WithFromEnv())
	if err != nil {
		return nil, err
	}

	traceExporter, err := otlptracehttp.New(ctx,
		otlptracehttp.WithEndpointURL(endpoint+"/v1/traces"),
		otlptracehttp.WithHeaders(headers))
	if err != nil {
		return nil, err
	}
	tracerProvider := sdktrace.NewTracerProvider(
		sdktrace.WithResource(res),
		sdktrace.WithBatcher(traceExporter))

	metricExporter, err := otlpmetrichttp.New(ctx,
		otlpmetrichttp.WithEndpointURL(endpoint+"/v1/metrics"),
		otlpmetrichttp.WithHeaders(headers),
		// Sherlock stores delta metrics. The SDK default is cumulative.
		otlpmetrichttp.WithTemporalitySelector(sdkmetric.DeltaTemporalitySelector))
	if err != nil {
		return nil, err
	}
	meterProvider := sdkmetric.NewMeterProvider(
		sdkmetric.WithResource(res),
		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter)),
		// Exemplars on every request, not only on sampled ones.
		sdkmetric.WithExemplarFilter(exemplar.AlwaysOnFilter),
		sdkmetric.WithView(sherlockView))

	otel.SetTracerProvider(tracerProvider)
	otel.SetMeterProvider(meterProvider)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{}))

	if err := runtime.Start(runtime.WithMeterProvider(meterProvider)); err != nil {
		return nil, err
	}

	return func(ctx context.Context) error {
		return errors.Join(
			tracerProvider.ForceFlush(ctx), meterProvider.ForceFlush(ctx),
			tracerProvider.Shutdown(ctx), meterProvider.Shutdown(ctx))
	}, nil
}

// durationBoundariesS is the bucket list Sherlock charts durations on, in seconds.
var durationBoundariesS = []float64{
	0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.25, 0.3,
	0.35, 0.4, 0.45, 0.5, 0.75, 1, 2.5, 5, 7.5, 10,
}

// sherlockView is the one view for every histogram: exemplars get trace_flags,
// and the two HTTP duration histograms get Sherlock's boundaries. Keep it one
// view; two views that match the same instrument export it twice.
func sherlockView(inst sdkmetric.Instrument) (sdkmetric.Stream, bool) {
	if inst.Kind != sdkmetric.InstrumentKindHistogram {
		return sdkmetric.Stream{}, false
	}
	stream := sdkmetric.Stream{
		Name:                              inst.Name,
		Description:                       inst.Description,
		Unit:                              inst.Unit,
		ExemplarReservoirProviderSelector: reservoirsWithTraceFlags,
	}
	switch inst.Name {
	case "http.server.request.duration", "http.client.request.duration":
		stream.Aggregation = sdkmetric.AggregationExplicitBucketHistogram{
			Boundaries: durationBoundariesS,
		}
	}
	return stream, true
}

// OTLP exemplars carry trace_id and span_id but not the sampled flag. The
// reservoir wrapper below adds trace_flags, "01" sampled or "00" not, to each
// exemplar, which tells Sherlock whether there is a trace to open.
func reservoirsWithTraceFlags(agg sdkmetric.Aggregation) exemplar.ReservoirProvider {
	provider := sdkmetric.DefaultExemplarReservoirProviderSelector(agg)
	return func(attrs attribute.Set) exemplar.Reservoir {
		return flagged{provider(attrs)}
	}
}

type flagged struct{ exemplar.Reservoir }

func (f flagged) Offer(
	ctx context.Context, t time.Time, v exemplar.Value, attrs []attribute.KeyValue,
) {
	if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
		flags := attribute.String("trace_flags", sc.TraceFlags().String())
		attrs = append(attrs[:len(attrs):len(attrs)], flags)
	}
	f.Reservoir.Offer(ctx, t, v, attrs)
}

What it sets, and why:

Setting	Why
`WithTemporalitySelector(DeltaTemporalitySelector)`	Sherlock stores delta metrics. The SDK default is cumulative, which also re-exports stale exemplars.
`WithExemplarFilter(exemplar.AlwaysOnFilter)`	The default `trace_based` filter drops exemplars for unsampled requests. Sherlock wants them, with `trace_flags` `00`.
`reservoirsWithTraceFlags`	OTLP exemplars carry `trace_id` and `span_id` but not the sampled flag. The wrapper adds `trace_flags` as a filtered attribute, so the UI knows which exemplars open a trace.
`sherlockView`	One view over every histogram. It injects the reservoir and gives the two HTTP duration histograms Sherlock’s 20 boundaries in seconds. It is one function on purpose: two views that match the same instrument export it twice.
`resource.WithFromEnv()`	Reads `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES`, so the service name and the `env` value stay out of the code.
`runtime.Start`	Seven `go.*` runtime metrics.

The file reads four variables:

export SHERLOCK_ENDPOINT='<endpoint>'     # Settings → Collector
export SHERLOCK_ACCESS_TOKEN='<token>'     # Settings → Collector, Reveal
export OTEL_SERVICE_NAME=checkout-api
export OTEL_RESOURCE_ATTRIBUTES=env=prod  # picks the source

Settings → Collector shows the Endpoint and the Bearer Token for your organization; copy both, since the endpoint differs by organization. env selects the source the data lands in, so use the value you chose in Get started. A value that matches no source is not shown.

sherlock.go

// sherlock.go: OpenTelemetry setup for Sherlock, configured through the
// standard OTEL_* environment variables. Only trace_flags needs code.
package main

import (
	"context"
	"errors"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/metric/exemplar"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

// Setup starts the traces and metrics pipelines. The exporters read the
// endpoint, the headers, and the temporality from the environment; the
// providers read the service name, the resource attributes, and the
// exemplar filter. Call the returned function after your server has stopped.
func Setup(ctx context.Context) (func(context.Context) error, error) {
	traceExporter, err := otlptracehttp.New(ctx)
	if err != nil {
		return nil, err
	}
	metricExporter, err := otlpmetrichttp.New(ctx)
	if err != nil {
		return nil, err
	}
	tracerProvider := sdktrace.NewTracerProvider(sdktrace.WithBatcher(traceExporter))
	meterProvider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter)),
		sdkmetric.WithView(traceFlagsOnExemplars))
	otel.SetTracerProvider(tracerProvider)
	otel.SetMeterProvider(meterProvider)
	otel.SetTextMapPropagator(propagation.TraceContext{})
	return func(ctx context.Context) error {
		return errors.Join(tracerProvider.Shutdown(ctx), meterProvider.Shutdown(ctx))
	}, nil
}

// traceFlagsOnExemplars is a view over every histogram that adds trace_flags,
// "01" sampled or "00" not, to each exemplar. OTLP has no field for the flag,
// so this is the one part of the setup that cannot come from the environment.
func traceFlagsOnExemplars(inst sdkmetric.Instrument) (sdkmetric.Stream, bool) {
	if inst.Kind != sdkmetric.InstrumentKindHistogram {
		return sdkmetric.Stream{}, false
	}
	return sdkmetric.Stream{
		Name:                              inst.Name,
		Description:                       inst.Description,
		Unit:                              inst.Unit,
		ExemplarReservoirProviderSelector: reservoirsWithTraceFlags,
	}, true
}

func reservoirsWithTraceFlags(agg sdkmetric.Aggregation) exemplar.ReservoirProvider {
	provider := sdkmetric.DefaultExemplarReservoirProviderSelector(agg)
	return func(attrs attribute.Set) exemplar.Reservoir {
		return flagged{provider(attrs)}
	}
}

type flagged struct{ exemplar.Reservoir }

func (f flagged) Offer(
	ctx context.Context, t time.Time, v exemplar.Value, attrs []attribute.KeyValue,
) {
	if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
		flags := attribute.String("trace_flags", sc.TraceFlags().String())
		attrs = append(attrs[:len(attrs):len(attrs)], flags)
	}
	f.Reservoir.Offer(ctx, t, v, attrs)
}

The exporters and providers take no options. Everything they need is in these six variables:

export OTEL_EXPORTER_OTLP_ENDPOINT='<endpoint>'   # Settings → Collector
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer <token>'
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
export OTEL_METRICS_EXEMPLAR_FILTER=always_on
export OTEL_SERVICE_NAME=checkout-api
export OTEL_RESOURCE_ATTRIBUTES=env=prod

Variable	What it does
`OTEL_EXPORTER_OTLP_ENDPOINT`	The Endpoint from Settings → Collector. The exporters append `/v1/traces` and `/v1/metrics`. Without it they send to `localhost:4318`.
`OTEL_EXPORTER_OTLP_HEADERS`	Headers as `key=value` pairs, comma-separated. Replace `<token>` with the Bearer Token from the same page.
`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE`	`delta`. Sherlock stores delta metrics; the SDK default is cumulative.
`OTEL_METRICS_EXEMPLAR_FILTER`	`always_on`, so unsampled requests get exemplars too, marked `trace_flags` `00`.
`OTEL_SERVICE_NAME`	`service.name`.
`OTEL_RESOURCE_ATTRIBUTES`	`env=<value>` routes the data into a source. See Get started.

What stays in code is trace_flags on exemplars: OTLP has no field for the sampled flag, so no variable can add it.This variant keeps the SDK defaults where the other one sets Sherlock’s values: otelhttp’s 14 histogram boundaries instead of Sherlock’s 20, no runtime metrics, and trace-context propagation without baggage. Each comes back with a few lines; see Histogram boundaries, Runtime metrics, and Baggage.Two mistakes stay quiet with this variant. With the endpoint variable unset, the SDK sends to localhost:4318 and prints this to stderr on every export:

traces export: Post "https://localhost:4318/v1/traces":
  dial tcp [::1]:4318: connect: connection refused

With the temporality variable unset there is no message at all: the data arrives cumulative and every chart climbs forever. Check the variables before the first run, and the Metrics page after it.

Wrap your server

Call Setup before the server starts, wrap the mux with otelhttp.NewHandler, and on shutdown stop the server first, then flush telemetry. This main.go is complete and works with either variant.

main.go

// main.go: a net/http service, instrumented by Setup from sherlock.go.
package main

import (
	"context"
	"log"
	"net/http"
	"os/signal"
	"syscall"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(),
		syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	shutdown, err := Setup(ctx)
	if err != nil {
		log.Fatal(err)
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /hello", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte(`{"hello":"world"}`))
	})
	mux.HandleFunc("GET /work", func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(150 * time.Millisecond)
		w.Write([]byte(`{"done":true}`))
	})

	// otelhttp.NewHandler wraps the mux, so every route is traced and metered.
	srv := &http.Server{
		Addr:              ":8080",
		Handler:           otelhttp.NewHandler(mux, "http.server"),
		ReadHeaderTimeout: 5 * time.Second,
	}
	go func() {
		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatal(err)
		}
	}()
	log.Print("listening on :8080")

	<-ctx.Done()
	closeCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
	defer cancel()
	// Finish in-flight requests first, then flush the last spans and metrics.
	if err := srv.Shutdown(closeCtx); err != nil {
		log.Print(err)
	}
	if err := shutdown(closeCtx); err != nil {
		log.Print(err)
	}
}

Register handlers with method-and-path patterns, such as GET /work. The span name and the http.route metric attribute come from the pattern. A handler registered as a bare path gives a span named GET with no route, and a mux mounted behind http.StripPrefix reports its mount pattern for every inner route. See Routers and nested muxes.

Run

go run .

Send a few requests so there is something to look at:

for i in 1 2 3 4 5; do
  curl -s localhost:8080/work
  curl -s localhost:8080/hello
done

Open Sherlock

Traces. Open Traces, switch to the Spans view, and pick checkout-api. Spans leave in batches of a few seconds. Each request from this main.go is one span named by its pattern, GET /work, with http.response.status_code on it. The Traces view lists only traces with more than one span, so it stays empty until a request makes a nested call.
Metrics. Open Metrics and choose http.server.request.duration. It appears after the first export, 60 seconds by default. Group by http.route to see the two routes apart. Each bucket carries an exemplar; click one to open the request’s trace.
Runtime. With the settings-in-the-file variant, go.goroutine.count and go.memory.used are on the same page.

Stop the app with Ctrl-C. The shutdown waits for the server to close, then flushes the last spans and metrics; both arrive.

What arrives

Data	Name	Unit
Server spans	`GET /work`, `GET /hello`, kind server, scope `otelhttp`
Request duration	`http.server.request.duration`, delta, 20 boundaries (14 with the environment variant)	`s`
Body sizes	`http.server.request.body.size`, `http.server.response.body.size`	`By`
Runtime	`go.goroutine.count`, `go.memory.used`, `go.memory.allocated`, `go.memory.allocations`, `go.memory.gc.goal`, `go.config.gogc`, `go.processor.limit` (settings-in-the-file variant)	various
Exemplars	one per non-empty histogram bucket per export, with `trace_id`, `span_id`, `trace_flags`

Outbound calls through an otelhttp.NewTransport client add http.client.request.duration with the same boundaries. Logs are not sent by this setup. Logs with trace ids adds them over OTLP with the same bearer token, or stamps stdout lines for a collector.

Troubleshooting

Nothing arrives

Export errors go to stderr through the SDK’s error handler, prefixed with traces export: or failed to upload metrics:. A 401 means the token is wrong or missing. Check that the endpoint and token variables are set in the shell that runs the service, that both match Settings → Collector, and that the endpoint has no trailing path.

Data arrives but the service is not in the source I expected

The env value in OTEL_RESOURCE_ATTRIBUTES did not match a source. Compare it with the source’s match value under Settings → Collector.

A span named GET with no route, or health checks in the metrics

A request matched no pattern, or a health checker hit the service. Requests that match no pattern get no http.route. Health checks need a filter; there is no default ignore list. See Skip health checks.

Metrics appear a minute late

The periodic reader exports every 60 seconds by default. Set OTEL_METRIC_EXPORT_INTERVAL=30000 for 30 seconds. The catalog refreshes every 60 seconds on top of that.

Configuration

Health checks, custom histograms, sampling, kill switches, logs, shutdown.

Differences from Node.js

What to expect if you know the Sherlock SDK for Node.js.

Traces

Find a request, read the waterfall, jump to its logs.

Exemplars

From a spike on a chart to the request behind it.

Send data

Node.js SDK

Go SDK

What you’ll learn

Prerequisites

Steps

What arrives

Troubleshooting

Configuration

Differences from Node.js

Traces

Exemplars

​What you’ll learn

​Prerequisites

​Steps

​What arrives

​Troubleshooting

​Related topics

Configuration

Differences from Node.js

Traces

Exemplars

What you’ll learn

Prerequisites

Steps

What arrives

Troubleshooting

Related topics