Logging & Observability (Prometheus, Grafana, OpenTelemetry)

devops/observability sitereliabilityengineering telemetry go videonotes

Core idea

Observability is not a tool you install. It is a property you build into the code: logs, metrics, traces, and profiles that let the system explain itself during an incident.

Hall’s test for every signal is the consumer. A log line, a metric, or a span earns its place only if a named person (developer, on-call engineer, support agent, auditor) can find it and act on it. The course applies that test to one small Go service, Linko, a URL shortener with deliberately planted bugs, and adds each layer only when a concrete question demands it.

Video

The frame: two questions and one toy service

The course opens with a failure story, not a definition. An e-commerce store returns 500 errors to 10 percent of visitors. The operator knows something is broken but cannot tell which requests trigger it, which of four servers is responsible, or whether the fault is regional.

Hall frames the whole subject as two questions:

1. How do I know when something is going wrong?
2. How do I find the problem so I can solve it quickly?

His claim about seniority follows from those questions. The difference is whether you build “a system that is able to help you debug itself” or carry all debugging work yourself. Passing a unit test suite is not the measure here.

The lab target is Linko, cloned from a starter repository and served on port 8899. The starter code contains intentional defects (“Easter eggs”) that the student must not fix early. You add instrumentation first, then use it to find the defects, which is the order of work in a real incident.

Course shape

The arc is: plain log package, then slog, then log strategy, error logging, context, storage, security, metrics, alerting, profiling, and tracing. Each chapter adds one signal type to the same service.

The observability model

Hall defines observability as the ability to know what a running application is doing, in real time and historically. He then uses the common three-pillar model and adds a fourth:

Signal	What it holds	Typical question it answers	Cost profile
Logs	Historical record of discrete events	What happened in this one request?	Grows with traffic, expensive to search at volume
Metrics	Aggregate numbers over time: requests per second, error count, CPU percent	Is something abnormal across thousands of events?	Small, cheap to store, low cardinality
Traces	A request’s path and timing across functions and services	Where did the time go, and in what order?	Sampled, moderate volume
Alerts	Proactive notification, most often driven by logs or metrics	Does a human need to act now?	Human attention is the scarce resource

The distinction he repeats is scale of attention: “logs tell you what happened during a specific event, but metrics help you see something weird that’s been happening across tens of thousands of events.”

Logs and metrics are complementary views of the same system, not competitors. Some facts belong in logs, some in metrics, some in both.

Related notes: Prometheus Introduction for the scrape model, and SLO, SLA and SLI for the targets that later justify alert thresholds. See “Prometheus Metrics” in the same folder for counters, gauges, and summaries.

Logging foundations in Go

Why a logger instead of fmt.Println

The argument is mechanical, not stylistic. Destination, timestamp, and formatting live in one place instead of thousands of call sites, so you can redirect all output without editing the application.

The standard log package supplies timestamps and a configurable writer, but it has no levels. log.Panic and log.Fatal look like levels and are not: they change what happens after printing, not the level printed.

Hall calls the stdlib syslog package effectively deprecated in practice. It is maintained because it is in the standard library, but he does not recommend it.

Practical defaults from the demo:

• Write to stderr, not stdout, as the convention for services.
• Do not reconfigure the global logger. Third-party libraries may depend on it, and parallel tests can interfere with each other.
• Inject the logger through constructors. Hall converts the Server and Store types to hold a logger field instead of reaching for a package-level global.

Fan out and bootstrap

Configurable output uses io.MultiWriter, which duplicates every write to several writers. The demo opens an optional log file from an environment variable and writes to both the file and stderr.

One exception to “never use fmt for logging”: logger construction itself. If the log file cannot be opened, no logger exists yet, so the failure goes straight to stderr with fmt.Fprintf. The same rule applies when closing the logger.

Buffering and the flush rule

Buffered writers batch disk writes for speed, at the cost of memory. The rule that follows is absolute in this course:

• Buffer file writers (8 KiB in the assignment).
• Never buffer stderr or stdout, or console output arrives in unpredictable bursts.

The demo bug is the one worth remembering: after several requests and a Ctrl+C, the log file is missing the most recent entries, because the buffer was never flushed. The fix is to return a close function from logger initialization and defer it.

Two calls that combine unrelated concerns

Hall’s position: log.Fatal and log.Panic “probably shouldn’t exist” and should “virtually never” be used.

Two reasons:

• They merge printing with control flow (exit or panic) in one call.
• log.Fatal skips deferred functions, so buffered logs are never flushed, files are never closed, and database connections are never released.

The replacement is explicit: log normally, return an exit code from a run function, and call os.Exit once at the top.

Deferred cleanup versus log.Fatal

Any buffered log data written before log.Fatal is lost. This is a data-loss bug, not only a style issue.

Structured logging with slog

Front end and back end

Structured logs store key/value pairs that can be parsed, stored, and queried in different formats. The benefit is precision of search: user_id=9284 matches essentially 100 percent, while a free-text search for 9284 also matches timestamps and file sizes.

slog separates two roles, which Hall compares to database/sql and its drivers:

• Logger: the front end your code calls.
• Handler: the back end that decides format and destination. TextHandler and JSONHandler ship with the standard library.

The second benefit follows from that split: you can change the format or destination without touching call sites.

Levels

Default levels and their integer values:

Level	Value	Hall’s guidance
Debug	-4	Events with little operational interest, including startup and shutdown
Info	0	Things you would search for later: successful logins, served requests
Warn	4	Mostly a gray area between Info and Error; often a sign nobody chose
Error	8	Something an administrator or developer must be told about

Two opinions stand out:

• Do not treat HTTP 4xx as errors. A 404 means the server worked and the client asked for nothing. “Virtually all the 400 series HTTP errors are not errors. They are user problems.” The quiz answer for a 404 response is Debug.
• Committed Debug logs usually do not belong. Ask whether a colleague enabling debug mode would want that line.

Fan out at the handler, not the writer

A multi-writer duplicates bytes but cannot filter by level. To send Debug and above to the console and Info and above to a file, the split must move up to the handler.

Hall states that Go 1.26 added slog.MultiHandler, which takes several handlers and fans each record out to all of them. Available documentation at the time of writing should be checked, since this is a recent standard-library addition.

Key/value discipline

Two equivalent syntaxes exist:

logger.Info("served request", slog.String("method", r.Method), slog.Int("status", 200))
logger.Info("served request", "method", r.Method, "status", 200)

Hall prefers typed attribute constructors. With bare alternating arguments, an odd count produces a malformed record where keys and values shift by one.

Text locally, JSON in storage

Hall’s rule: human-readable text on the console for local debugging, JSON in the file or log store for machines. Switching only the file handler to JSONHandler is a small change and keeps the local experience readable.

Log strategies

Name the consumer first

Before writing a line, identify who reads it:

• Developers: maintenance and debugging. The risk is tunnel vision.
• Operators and SRE: resource use, startup and shutdown failures, operational state.
• Customer support: business questions such as why a login failed.
• End users: apps that expose logs, needing different wording and granularity.
• Auditors: compliance questions of who did what, when.

These groups may need separate streams, for example an operational log, a customer-facing log, and an audit log.

Discoverability, clarity, specificity

“Logs that you cannot discover might as well not exist.” The practical corollary: too much detail beats too little, because missing data cannot be recovered after the incident.

Specificity means answering where, why, when, who, and which server. “File not found” fails that test on its own.

Logs are not free

Storage is cheap, but attention and search cost are not. His reduction rules:

• Do not log the same event in the handler, the middleware, and the service layer.
• Do not log events that may resolve themselves. In a retry loop of five attempts, log the fifth failure, not five failures.
• Use levels so Debug costs nothing in production when Debug is disabled.
• Structure the fields so the record is findable.

One log per event

Two anti-patterns appear:

• Ad hoc tracing: “calling get user from DB”, then “entering get user from DB.” Useful when no debugger is available, unwanted in production.
• Duplicated failure logs at inner and outer layers. The exception Hall allows is when the inner log adds a field the outer log cannot know, such as the index of the failing cart item.

The worked example is a PID-file read with four progress logs. It collapses to one log per outcome, because each failure implies the previous steps succeeded.

Timestamps and tests

Timestamps are the one field that is always relevant. They are also the field that breaks tests.

The technique is slog.HandlerOptions.ReplaceAttr to rewrite the time attribute to a fixed value. Hall’s practice: assert the timestamp once per logger to prove the configuration, then omit time from most assertions.

Error logging

Errors are values, not events

Go errors are a data type. A file-not-found value does not prove the system misbehaved, and some errors are handled silently on purpose. Hall’s Venn diagram: most errors are logged, some logs are errors, and a region of ignored errors is never logged.

Stack traces

He adds stack traces to errors “almost universally”, for one reason: “failed to connect to database” is useless when thousands of call sites can produce it, while a trace names the function and line.

Go’s standard library does not attach traces. The practical option is github.com/pkg/errors, imported under an alias to avoid confusion with the standard errors package. pkgerrors.WithStack adds the trace and %+v renders it.

Handle each error once

The guiding principle comes from Dave Cheney: handle errors only once. Handling means inspecting the value and deciding something. Logging and returning is two decisions about one failure.

The demonstration is an order validation path where both the inner validator and the outer caller log the same failure. The improved form wraps with context at the inner layer and logs only at the outer layer:

return fmt.Errorf("failed to validate item %d (%v): %w", itemNumber, description, err)

Hall is careful about the trade-off: one log line instead of two, but the extra fields are no longer structured.

Structured errors with slog.Group

To recover structure, he defines an error type that carries []slog.Attr, plus a helper that walks the wrap chain and collects attributes.

Two details worth copying:

• The type must implement Unwrap() error so errors.Is and errors.As keep working.
• Walking the chain manually is correct here even though Go guidance prefers errors.Is. You want every match, not the first.

ReplaceAttr then turns any error attribute into a group with message, optional stack_trace, and the collected attributes.

Multiple simultaneous failures

errors.Join merges several errors from batch or fan-out work. His caveat matters more than the code: there is “no obviously correct way” to render nested multi-errors, so solve for the shapes your application actually produces and avoid nesting them.

Linko’s list operation changes behavior as a result: it collects failures and returns partial results instead of stopping at the first error. Hall judges that “strictly better” only when aborting early is not required.

Logging context

Hall’s framing is that logging is storytelling: the same event means different things depending on setting, time, and participants.

The interview exercise is a single line: level=error message="failed to read file" error="permission denied". The missing context is timestamp, file name, and user identity, because “if 500 users made requests last night, but only two of them got 500s, I need to know which one”.

Four context layers, each answering a different question:

Layer	Question	Example fields	When it matters
Build	Which code is running?	git SHA, build time, version	Bug reports that arrive days after a deploy
Instance	Which copy of the process?	environment, hostname, region, pod name, kernel	Failures on one pod or one environment
Request	Which interaction?	method, path, status, duration, body size	Reproducing one failing interaction
User	Whom did it affect?	user name or ID	Scoping blast radius to people, not servers

Build context with linker flags

Two mechanisms exist. go build -buildvcs plus debug.ReadBuildInfo reads version control metadata at runtime, but it requires the repository inside the build context, which bloats container images.

Hall prefers linker flags:

go build -ldflags "-X my/pkg.gitSHA=$(git rev-parse HEAD) -X my/pkg.buildTime=$(date -u +%Y-%m-%dT%H:%M:%SZ)"

Defaults of "unknown" make a missing injection visible at runtime. The flags are attached once in the logger constructor, so every record inherits them.

ldflags do not apply to go run

You must build a binary to test the injection. Hall calls this “the easiest to forget” step and recommends a Makefile or CI job.

Request context without hand-written plumbing

Measuring duration is simple: capture the start time in middleware and log slog.Duration.

Body and response sizes need small spies that wrap io.ReadCloser and http.ResponseWriter. Two honest limitations:

• The request spy counts bytes read, not bytes present. An ignored body logs zero, and a partial JSON parse counts only the consumed prefix.
• The response spy must reproduce net/http defaults: a Write without WriteHeader implies 200, and an empty response leaves both values zero.

User context across middleware

The interesting problem is ordering. The logger middleware sits above the auth middleware, so the user name becomes known after the logger has started its work.

The solution is a shared mutable struct in the request context:

1. The outer request logger allocates &LogContext{} and stores the pointer in the context.
2. The inner auth middleware writes logCtx.Username through that pointer.
3. After serving, the logger reads the field and adds a user attribute only when it is non-empty.

A value instead of a pointer silently loses the writes. A shadowed logger variable inside the closure caused a 401 response to log the previous user; the fix was to capture a local lg:= logger per request.

Request IDs

The transport mechanism for context across process boundaries is a unique ID that travels with the request. Linko’s gateway behavior is the useful pattern:

• Accept an inbound X-Request-ID when present, otherwise generate one.
• Echo it back in the response header so the client can quote it in a bug report.
• Register this middleware before the request logger, which then attaches the ID to every record.

Do not trust an inbound ID blindly. A client may send none, or send a stale value from an old SDK.

Log storage

Four destinations

Destination	Fit	Note
Console	Containers, development	Often the real source: an agent reads stdout and ships it
File	Single-host services, small scale	Rotation and disk cleanup become your problem
Cloud log service	Distributed systems, fast search	Moves permission management rather than removing it
Self-hosted stack	Full control, real cost	You operate Elasticsearch or equivalent

Rotation

Value decays with age, so the live file rotates. The naming cascade runs linko.log, linko.log.0, linko.log.1, and so on, with older generations compressed.

The demo uses lumberjack, a plain io.Writer with maximum size, maximum age, backup count, and compression settings. Hall notes that most services use the operating system’s logrotate instead, and that lumberjack suits single-binary Go deployments.

syslog

syslog is a 1980s protocol that is still widely implemented and slowly declining. The key limitation for this course: Go’s standard log/syslog package does not do structured logging, even though the protocol supports structured data. Use an slog adapter instead.

Colour on the console only

tint replaces the text handler and colours by level. Gate it on TTY detection so redirected output carries no escape codes. Colour is a development aid; the aggregator can add its own.

Log security

The prohibited list

Obvious: passwords, API keys, credit card numbers, social security numbers.

Less obvious, and the focus of the chapter:

• Account numbers and the existence of other users in the system.
• Internal architecture, table names, hostnames.
• Personal data: names, addresses, phone numbers, and in many jurisdictions IP addresses, which he calls “potentially dangerous from a PII standpoint.”
• Unreleased business material, if logs are not tightly controlled.

Privacy is not binary. Passwords should be stored only as hashes; card data belongs with a payment provider; an internal table name is low risk.

The DSN case

Logging a database connection string from an environment variable is the worked example. A typical DSN embeds a username, a password, and a database name. The password must never appear. The username and database name may help debugging in multi-tenant systems, and should be omitted when they do not. The mitigation is to log a sanitized copy with the credential removed.

Error text as an information leak

Two real leaks appear:

• sql: no rows in result set reveals that a user name does not exist, which enables user enumeration.
• table migrated_users does not exist reveals schema and, worse, that deployments may be sloppy, which invites further probing.

His rule: replace all authentication and authorization failures with one generic message. Linko maps 401, 403, and 500 bodies to the lowercase standard status text, while the full error goes to the log.

He acknowledges the usability cost: the user can no longer tell “no such account” from “wrong password.” He prefers enumeration safety and suggests email-based recovery flows for revealing account existence.

Different consumers, different detail

The operator sees table migrated_users does not exist. The user sees internal server error. Both are correct for their audience.

The full request and response anti-pattern

Logging an entire request URL, body, and response body to debug a third-party API is common and dangerous. Risks:

• The API key itself sits in the log, and logs are not in a secret manager.
• The response may contain card data or health records that the developer should not see at all.

The safer version he uses: unmarshal the response and log only the JSON keys, optionally with value types, which shows shape without content.

Even method, path, and status are not automatically safe. A third-party API that accepts the key as a query parameter puts the secret in the path.

Redaction in practice

Linko redacts the last IPv4 octet (127.0.0.1 becomes 127.0.0.x) and truncates the last IPv6 segment. The helper uses net.SplitHostPort and net.ParseIP, and returns the input unchanged when parsing fails, because this is a logging filter and not a security boundary.

Filtering after the fact is the last line of defence, not the control. The ReplaceAttr filter replaces values of sensitive keys (password, api_key, secret, pin, credit_card, user) with the literal redacted, and keeps the key so you can see that someone tried to log it.

Encryption is the extreme option. Hall used the age tool once or twice in his career for a payment API that had to be logged. It makes records larger, not smaller, and should be reserved for data you “absolutely must log” with a need to recover it.

Metrics

Three shapes

Type	Behavior	Example
Counter	Only increases, resets on restart	Requests served, failed logins since start
Gauge	Rises and falls	Percent CPU, open connections, free bytes
Histogram	Buckets a distribution	Response time: 80 percent under 5 ms, 95 percent under 10 ms

A percentile such as p95 over five minutes is a histogram question, not a gauge.

Prometheus is a pull-based store

Hall’s framing: think of Prometheus as a database, specifically a time-series database, not a general store.

Most databases wait for writes. Prometheus polls targets that it learns from configuration, which is why the model is called a pull model.

The incident drill that motivates system metrics: a 3 a.m. page for sporadic 500s, caused by a full disk writing session files. His answer is to alert above 80 percent utilization or below 10 GB free, and better, to alert when the rate of consumption changes.

What to measure

Two frameworks, both worth memorizing:

• RED for request-serving services: Rate of events per second, Errors, Duration. Rate and errors are counters; duration is a histogram.
• USE for resources: Utilization, Saturation, Errors. This maps directly onto Node Exporter data.

The RED histogram example is concrete: 1567 successes against 321 failures, with 432 requests under 1 second and a tail out at 5 seconds. If the tail is unacceptable, the fix is to move the distribution left, not to alert on it.

Instrumenting the Go service

The client library is github.com/prometheus/client_golang, and promhttp.Handler serves the /metrics endpoint that Prometheus scrapes. Go runtime metrics come for free.

Custom metrics use a CounterVec with labels. Labels are the dimensions of a metric and the main source of cost, because every label combination is a separate time series:

var httpRequestsTotal = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "total number of HTTP requests",
    },
    []string{"method", "path", "status"},
)

The increment sits in middleware, the “choke point” every request passes through, and reuses the response spy from the logging middleware. Two details:

• Status codes must be converted to strings for string labels.
• If a handler never sets a status, default to 200 with cmp.Or.

Grafana panels that tell the truth

Prometheus serves its query UI on port 9090 and acts as the data source; Grafana reads it over the Compose network. The community dashboard for host metrics is ID 1860, “Node Exporter Full.”

Two lessons from building the request panel:

• A raw counter rises forever and resets on restart, so it is a bad dashboard. Wrap it in rate().
• Distinguish the display interval from the rate window. Sampling a one-minute rate every five seconds gives repeated flat ticks; matching the two smooths the line.

The status panel uses increase(http_requests_total[...]) grouped by (status), then a reduce transformation to collapse series into one bar per code.

Alerting

Actionable or nothing

Logging and metrics are passive. Alerting is the layer that tells a human to act.

The demonstration is deliberately bad: a rule that fires on a spike of 401 logins. Failed logins are normal on a public site. The consequence is alert fatigue, where you silence your phone and then miss a real page.

The fix is a change of subject, not a change of threshold: alert on system errors (500s), not user errors (401s). The demo shows the asymmetry: a wrong password triggers nothing, while a malformed password that produces a 500 fires the rule.

Three numbers define a threshold

Number	Question	Effect
Threshold value	How many failures are enough?	Too low creates noise
Query window	Over what period?	Too short creates flapping
Pending period	How long must it stay above?	Zero means every spike pages someone at 3 a.m.

The lab sets a 20-second pending period and shows the rule passing through a pending state before firing.

Beyond this chapter

Contact points, routing, paging versus ticketing, and runbooks are mentioned only as configuration fields. The course does not cover escalation policy in depth.

Profiling

Types and when to use them

Profiling measures where a running program spends resources:

• CPU: which functions consume processor time.
• Memory or heap: which functions allocate.
• Goroutine: which code creates goroutines and how long they live.
• Blocking: time spent waiting on I/O or locks.
• Execution tracing: a timeline of what ran, in order.

Hall’s answer to “when should I profile” is blunt: “you should profile when you discover a problem and not before.” The exception is a very small number of performance-critical functions, such as a crypto primitive whose speed bounds throughput.

That is the discipline of profile-driven development: optimize the hotspot the profile names, not the code that looks slow.

Exposing pprof without exposing it to the internet

net/http/pprof is in the standard library. A blank import registers handlers on the default mux; an application with its own mux must register them explicitly.

The security guidance is explicit: never expose pprof on the public port, because “all of your SaaS clients shouldn’t have access to your pprof data.” Options named: a private port behind the VPN, an auth middleware, or enabling it only in development.

The CPU walkthrough

Profiling is not always running. The request starts a capture for N seconds and returns the result:

curl --user user:pass localhost:8899/debug/pprof/profile?seconds=30 -o cpu.prof
go tool pprof cpu.prof

Results from the demo: 11.08 seconds of the 30-second window were busy, 36.93 percent. A single function, blowfish.encryptBlock, held 10.56 seconds, about 95 percent of CPU time.

Reading flat, cumulative, and the graph

View	Meaning
flat	Time attributed to the function itself
cum	Time in the function plus everything it calls

The graph view makes the call chain visible: each box is a function, box size is cost, and arrows show call order. The value is the path back into your own code.

go tool pprof -dot cpu.prof | dot -Tsvg > cpu.svg

The heap walkthrough

The heap profile found makeslice inside strings.Builder, reached from handlerRedirect, growing to about 3.5 MB in a 30-second window. The cause was a handler appending a repeated URL 1,024 times to a package-level slice.

Hall distinguishes excessive allocation from a true leak: the slice grows, but it is still reachable.

Goroutines are harder

The goroutine endpoint takes no duration, because it is a single snapshot rather than a window. Health is judged by where goroutines are parked: many in runtime.gopark and selectgo is normal, many inside one handler is not.

Goroutine leaks are hard to reproduce because goroutines are cheap: they consume no CPU while idle and very little memory, so a leak may need millions of requests to become visible.

Hall mentions that a recent Go release added goroutine leak profiling support. The version number in the captions is unclear, so check the current release notes before relying on it.

Tracing

The vocabulary

Term	Definition
Trace	One request, end to end, possibly across services
Span	One unit of work in the trace, with a name, ID, start, and end
Attributes	Key/value pairs on a span, like structured log fields
Events	Timestamped annotations inside a span
Status	unset, ok, or error

A span is like a stack frame but not one-to-one. One span may cover a database call that passes through several functions, so a trace usually has fewer spans than the call stack has frames.

Logs versus traces

His comparison is the clearest statement in the chapter: a log is a single event in time, while a trace is a two-dimensional view of several events, their order, and their durations.

He is also honest that the distinction is representational, not informational: “conceptually nothing a trace can provide that a log cannot.” You could emit one log per step with durations and a request ID, but the result is a crowded record that you would end up visualizing anyway.

Setup: OpenTelemetry plus Jaeger

The split mirrors Prometheus and Grafana: OpenTelemetry captures and exports, Jaeger displays.

initTracing does five things:

1. Create an OTLP gRPC exporter.
2. Create a TracerProvider with it.
3. Set a 2-second batch timeout.
4. Set the global tracer provider.
5. Return the provider’s shutdown function.

Defaults target localhost:4317 with TLS. For local development, disable TLS and set the service name through environment variables.

Always call the shutdown function

Without defer provider.Shutdown(...), the last spans never reach Jaeger. This is the same class of bug as the unflushed log buffer. Add a timeout (5 or 30 seconds) so shutdown cannot block exit.

Root span and child spans

otelhttp.NewHandler wraps the mux and creates a root span for every inbound request, injecting it into the request context.

Child spans take two lines per handler:

ctx, span := tracer.Start(r.Context(), "handler.shortenLink")
defer span.End()

The tracer is a package-level variable created from the provider with a scope name such as boot.dev/linko, which groups spans by source.

What deserves a span, in his judgment:

• Yes: HTTP handlers, and middleware that does real work such as authentication and its user lookup.
• No: middleware that runs on every request, because its presence is already implied by the trace.

Context discipline

This is the main gotcha. Once tracer.Start returns a new context, every downstream call must receive that context, not r.Context, or child spans attach to the wrong parent.

His habit: write ctx:= r.Context at the top of every handler, so adding tracing later does not require rewriting each call.

Reading the demo trace

The shorten request took 124.21 ms in the screenshot and 112 ms in the live demo.

flowchart TD
  A["HTTP handler (root)"] --> B["handler.shortenLink"]
  B --> C["auth.validatePassword<br/>bcrypt, intentionally slow"]
  B --> D["http.verifyDestination<br/>outbound request"]

Synthesized span tree, simplified from the demo trace.

Two readings matter:

• validatePassword is slow by design. bcrypt is deliberately expensive so an attacker cannot test millions of passwords, and a legitimate user does not notice.
• shortenLink holds the business logic but its active work is tiny. Most of its duration is waiting on verifyDestination, an outbound call that checks the target URL.

That is the diagnostic value of a trace: it separates time spent computing from time spent waiting, which a log line cannot show.

Distributed tracing

Across services, the mechanism is the trace ID that tracer.Start generates and stores in the context.

For HTTP, the ID travels in a header. The transport that does this is otelhttp.NewTransport, wrapped around the HTTP client so the outbound request carries the trace context:

client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

The receiving service must use the same context to continue the trace, and OpenTelemetry SDKs in other languages implement the same propagation, so traces cross language boundaries.

The closing quiz answer summarizes it: spans from different services join into one trace because trace context, including the trace ID and parent span ID, is propagated between services, typically in HTTP headers.

How the pieces fit

flowchart TD
  App["Linko (Go)"] --> L["Logs: slog JSON"]
  App --> M["/metrics"]
  App --> P["/debug/pprof"]
  App --> T["Spans: OTLP"]
  L --> Agg["Log store / aggregator"]
  M --> Prom["Prometheus (pull)"]
  T --> Jaeger["Jaeger"]
  Prom --> Graf["Grafana"]
  Prom --> Alerts["Alert rules"]
  P --> Prof["CPU / heap profiles"]

Synthesized map of the course architecture. It is not a diagram from the video.

A decision table for the next incident:

Question	Signal to reach for
What exactly happened in this request?	Log with request and user context
Is this widespread or isolated?	Metric rate or error ratio
Where did the time go?	Trace
Why is this function slow or allocation-heavy?	CPU or heap profile
Should a human wake up?	Alert with a pending period

Key takeaways

1. Observability is a property of the code, not a vendor. Instrument the service so it answers questions later.
2. Name the consumer of every signal. Developer, on-call engineer, support agent, auditor, and end user need different detail and different streams.
3. Structured logging is the foundation. Key/value pairs make search precise, and slog separates formatting from the logging call.
4. Log each event once, at the layer that can act on it. Log-and-return is the anti-pattern that produces six copies of one failure.
5. Context turns an event into a diagnosis. Attach build, instance, request, and user fields, and propagate a request ID across boundaries.
6. Never log secrets or personal data. Redact, filter as a backstop, and encrypt only when recovery is genuinely required.
7. Metrics show patterns that logs hide. Use counters, gauges, and histograms, and pick RED for services or USE for resources.
8. Alerts must be actionable. Use 500s, not 401s, and set value, window, and pending period together.
9. Profile only when a problem exists. Read flat, cumulative, and the call chain back to your own code.
10. Traces explain time and order. Separate computing from waiting, and propagate context so one trace spans services.

Action checklist for a Go service

• Replace fmt.Print logging with an injected slog logger writing to stderr, plus JSON to a file or store.
• Set build fields through ldflags and attach them once in the logger constructor.
• Add environment, hostname, request ID, and user fields through a shared context struct.
• Flush buffered file logs on shutdown, and never buffer stderr.
• Remove log-and-return pairs; wrap with %w and log once at the top.
• Add stack traces to error logs and group the fields.
• Redact IP addresses, credentials, and user identifiers; filter sensitive keys as a backstop.
• Mount /metrics, then add a CounterVec for method, path, and status.
• Build a rate() panel and a by-status bar panel in Grafana.
• Write one alert on 5xx with a pending period, and send it to a real contact point.
• Expose pprof only behind auth or a private port.
• Wrap the mux with otelhttp, add spans to handlers, and shut the provider down with a timeout.

Source limits

This note is built from the automatic English captions only. No slides or screen contents were available, so code shown on screen is reconstructed from narration and may differ in detail. Product names and numbers that the captions render ambiguously were corrected only where context was clear. Version claims, such as slog.MultiHandler arriving in Go 1.26 and new goroutine leak profiling, are the presenter’s statements at recording time and were not checked externally. Timing and CPU figures are demo-observed and approximate.