0Pricing
gRPC & High Performance APIs · Lesson

Tracing with OpenTelemetry

Integrate OpenTelemetry to achieve distributed tracing, visualizing end-to-end request flows across services.

Tracing with OpenTelemetry is a free gRPC & High Performance APIs lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the gRPC & High Performance APIs learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Microservices & The Tracing Need

Modern applications often consist of many small, interconnected services, known as microservices. This architecture offers flexibility but can make debugging complex interactions a real challenge.

When a user request travels through several services, pinpointing where a delay or error occurred becomes like finding a needle in a haystack.

Following the Request Path

Distributed tracing helps you visualize the entire journey of a request as it flows through different services in your system.

  • It links all operations related to a single request.
  • It shows the time taken by each part of the request.
  • This helps identify performance bottlenecks and errors across service boundaries.

OpenTelemetry: The Observability Standard

OpenTelemetry (OTel) is an open-source project that provides a standardized way to collect telemetry data from your applications. This includes traces, metrics, and logs.

Instead of vendor-specific SDKs, OTel offers a single set of APIs, SDKs, and tools to instrument your services, making your observability data portable.

Traces Built from Spans

In OpenTelemetry, a trace represents the complete execution path of a request through your system. It's like the entire story of that request.

A trace is composed of one or more spans. Each span represents a single operation or unit of work within that trace, like a step in the story. Spans have a start time, end time, and metadata.

Propagating Trace Context

For spans to form a complete trace across different services, they need to be linked. This is done through context propagation.

When a service calls another, it passes along a "trace context" (often in HTTP/2 headers for gRPC). This context tells the receiving service which trace and parent span its new operations belong to, creating a parent-child relationship between spans.

OTel Tracing Essentials

To implement tracing, you'll work with these core OpenTelemetry components:

  • TracerProvider: Manages Tracer instances and configures span processors and exporters.
  • Tracer: Creates Span objects.
  • SpanProcessor: Processes spans before they are exported (e.g., batching).
  • Exporter: Sends collected spans to a backend (like a console, Jaeger, or OTLP collector).

OTel Setup: TracerProvider & Exporter

Let's set up a basic OpenTelemetry TracerProvider in Go. We'll use a simple stdouttrace exporter to print trace data to the console, so you can see the spans being generated.

This snippet initializes the global TracerProvider and ensures all spans are flushed when the application exits.

Run this code to see the basic setup:

package main

import (
	"context"
	"log"
	"os"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
	"go.opentelemetry.io/otel/trace"
)

// initTracerProvider initializes an OpenTelemetry TracerProvider
// with a stdout exporter for demonstration.
func initTracerProvider() *sdktrace.TracerProvider {
	// Create stdout exporter to print traces to the console
	exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
	if err != nil {
		log.Fatalf("failed to create stdout exporter: %v", err)
	}

	// Create a new TracerProvider with the exporter and a service name
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter), // Batch spans for efficiency
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName("my-simple-service"),
			attribute.String("environment", "development"),
		)),
	)
	otel.SetTracerProvider(tp) // Set the global TracerProvider

	return tp
}

func main() {
	tp := initTracerProvider()
	defer func() {
		if err := tp.Shutdown(context.Background()); err != nil {
			log.Fatalf("Error shutting down tracer provider: %v", err)
		}
	}()

	// Get a tracer
	tracer := otel.Tracer("my-app-tracer")

	// Create a root span
	ctx, span := tracer.Start(context.Background(), "main-operation")
	defer span.End()

	log.Println("Hello from main-operation with tracing enabled!")
	time.Sleep(100 * time.Millisecond) // Simulate some work

	// Create a child span
	_, childSpan := tracer.Start(ctx, "sub-operation")
	defer childSpan.End()
	childSpan.SetAttributes(attribute.String("task", "processing"))
	log.Println("Doing some sub-operation...")
	time.Sleep(50 * time.Millisecond) // Simulate more work

	log.Println("Trace data will be printed on shutdown.")
}

Auto-Tracing gRPC Calls

To automatically trace gRPC requests, OpenTelemetry provides interceptors. These are middleware functions that wrap gRPC calls on both the client and server sides.

  • Server interceptor: Starts a new span for incoming requests, linking it to the trace context from the client.
  • Client interceptor: Injects the current trace context into outgoing requests, allowing propagation to the server.

Here's how you'd apply them to your gRPC server and client:

package main

import (
	"context"
	"log"
	"net"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	// OpenTelemetry imports
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"

	// gRPC OpenTelemetry instrumentation
	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

	// For this example, we'll define a simple service in-line
	pb "google.golang.org/grpc/examples/helloworld/helloworld" // Using a standard example proto
)

// initTracerProvider initializes an OpenTelemetry TracerProvider
// with a stdout exporter for demonstration.
func initTracerProvider() *sdktrace.TracerProvider {
	exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
	if err != nil {
		log.Fatalf("failed to create stdout exporter: %v", err)
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName("grpc-otel-demo"),
			attribute.String("environment", "development"),
		)),
	)
	otel.SetTracerProvider(tp)
	return tp
}

// helloServer is a simple gRPC server for demonstration
type helloServer struct {
	pb.UnimplementedGreeterServer
}

func (s *helloServer) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
	log.Printf("Received: %v", in.GetName())
	// Simulate some work
	time.Sleep(50 * time.Millisecond)
	return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}

func main() {
	// 1. Initialize OpenTelemetry TracerProvider
	tp := initTracerProvider()
	defer func() {
		if err := tp.Shutdown(context.Background()); err != nil {
			log.Fatalf("Error shutting down tracer provider: %v", err)
		}
	}()

	// 2. Start gRPC Server with OpenTelemetry Interceptor
	lis, err := net.Listen("tcp", ":50051")
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	s := grpc.NewServer(
		grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()), // Apply OTel server interceptor
	)
	pb.RegisterGreeterServer(s, &helloServer{})
	log.Println("Server listening on :50051")
	go func() {
		if err := s.Serve(lis); err != nil {
			log.Fatalf("failed to serve: %v", err)
		}
	}()
	defer s.Stop()

	// Allow server to start
	time.Sleep(1 * time.Second)

	// 3. Create gRPC Client with OpenTelemetry Interceptor
	conn, err := grpc.Dial(
		"localhost:50051",
		grpc.WithInsecure(), // For simplicity, use insecure
		grpc.WithBlock(),
		grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()), // Apply OTel client interceptor
	)
	if err != nil {
		log.Fatalf("did not connect: %v", err)
	}
	defer conn.Close()
	c := pb.NewGreeterClient(conn)

	// 4. Make a gRPC call
	log.Println("Making gRPC call...")
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	defer cancel()

	r, err := c.SayHello(ctx, &pb.HelloRequest{Name: "CoddyKit User"})
	if err != nil {
		log.Fatalf("could not greet: %v", err)
	}
	log.Printf("Greeting: %s", r.GetMessage())

	// Give some time for spans to be processed and exported
	time.Sleep(1 * time.Second)
	log.Println("Trace data should now be visible in console.")
}

Reading the Trace Story

After running the previous example, you'll see a detailed JSON output in your console. This output represents the trace data.

  • Look for "TraceID": This unique ID links all spans belonging to the same request.
  • Each entry is a "Span": It has a "SpanID", "ParentSpanID" (if it's not the root), name, start/end times, and attributes.
  • The hierarchy of spans (parent-child) shows the flow of execution.

This "story" helps you understand how long each step took and which service was responsible.

Enriching Spans with Attributes

While auto-instrumentation provides basic spans, you often need to add more specific information to your traces. Span attributes (key-value pairs) allow you to do this.

You can record details like user IDs, request parameters, or specific business logic outcomes directly on a span. This makes debugging and analysis much more effective.

Use span.SetAttributes() to add custom attributes.

package main

import (
	"context"
	"log"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
	"go.opentelemetry.io/otel/trace"
)

// initTracerProvider initializes an OpenTelemetry TracerProvider
func initTracerProvider() *sdktrace.TracerProvider {
	exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
	if err != nil {
		log.Fatalf("failed to create stdout exporter: %v", err)
	}
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName("custom-attributes-demo"),
		)),
	)
	otel.SetTracerProvider(tp)
	return tp
}

func main() {
	tp := initTracerProvider()
	defer func() {
		if err := tp.Shutdown(context.Background()); err != nil {
			log.Fatalf("Error shutting down tracer provider: %v", err)
		}
	}()

	tracer := otel.Tracer("my-custom-tracer")

	// Start a root span
	ctx, span := tracer.Start(context.Background(), "process-order")
	defer span.End()

	// Add custom attributes to the root span
	span.SetAttributes(
		attribute.String("user.id", "user123"),
		attribute.Int("order.id", 45678),
		attribute.Bool("is_premium_user", true),
	)
	log.Println("Processing order with custom attributes...")
	time.Sleep(100 * time.Millisecond)

	// Create a child span
	_, childSpan := tracer.Start(ctx, "database-query")
	defer childSpan.End()
	childSpan.SetAttributes(
		attribute.String("db.system", "postgres"),
		attribute.String("db.statement", "SELECT * FROM orders WHERE id=45678"),
	)
	log.Println("Executing database query...")
	time.Sleep(50 * time.Millisecond)

	log.Println("Check the console output for custom attributes in the spans!")
	time.Sleep(500 * time.Millisecond) // Ensure spans are exported
}

Tracing Concepts Check

You've learned about distributed tracing and how OpenTelemetry helps. Let's test your understanding.

Tracing with OpenTelemetry Recap

In this lesson, you learned how distributed tracing helps navigate the complexities of microservices by showing the full journey of a request.

We explored OpenTelemetry as the standard for collecting traces, understanding core concepts like traces, spans, and context propagation. You saw how to set up an OTel TracerProvider and how gRPC interceptors make instrumenting your services straightforward. Finally, we covered enriching your spans with custom attributes for deeper insights.

With OpenTelemetry, you gain crucial visibility into your gRPC application's behavior!

Frequently asked questions

Is the “Tracing with OpenTelemetry” lesson free?

Yes — the full text of “Tracing with OpenTelemetry” is free to read here on the web, and the gRPC & High Performance APIs course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the gRPC & High Performance APIs course, upgrade to CoddyKit PRO.

What will I learn in “Tracing with OpenTelemetry”?

Integrate OpenTelemetry to achieve distributed tracing, visualizing end-to-end request flows across services. You practise gRPC & High Performance APIs with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start gRPC & High Performance APIs?

No prior experience is required. gRPC & High Performance APIs on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Tracing with OpenTelemetry” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this gRPC & High Performance APIs lesson?

Yes. Every gRPC & High Performance APIs lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Logging gRPC Interactions
  2. Tracing with OpenTelemetry
  3. Monitoring gRPC Metrics
  4. Health Checking & Readiness Probes
← Back to gRPC & High Performance APIs