Reading Go goroutine dumps by state, stack, and wait time

Go services fail in ways logs do not explain. A process is up, CPU is low, and requests still hang. Memory climbs, or a deploy never drains. In those moments, a goroutine dump is one of the few artifacts with direct evidence of what the runtime is doing.

The problem is volume. A SIGQUIT dump from a busy process is long and repetitive. Hundreds or thousands of goroutines differ only in one wait site. You need grouping, counts, and wait-time clues before you need another graph. Go Goroutine Dump Analyzer — read a SIGQUIT stack dump focuses on this step. It groups a pasted dump by state and by stack, surfaces wait durations, and highlights stacks held by far more goroutines than they should be.

For a practitioner, this matters because goroutine failures often look like application failures, network failures, or database failures from the outside. The dump gives you a runtime view. Which goroutines are blocked on channels. Which are stuck in I/O wait. Which are parked on mutexes or semaphores. Which stacks repeat across a suspicious share of the process. Read well, this data narrows your search fast.

Start with shape, not individual frames

The first useful question is simple. What shape does the dump have?

A healthy process often has a long tail of distinct stacks and a few expected clusters, HTTP server workers waiting in poll, background timers, GC helpers, and client pools. A troubled process often has a dominant cluster. Many goroutines share one stack prefix and one wait state. That pattern points to contention, fan-out without backpressure, a leaked worker path, or a downstream dependency holding callers open.

Grouping by state helps you sort this fast. In Go dumps, common states include chan receive, chan send, select, IO wait, semacquire, sync.Cond.Wait, and runnable or running states. Each state narrows the class of fault.

• A large chan send group points at producers with no progress on the consumer side.
• A large chan receive group points at idle consumers, starvation upstream, or a shutdown path waiting on work that never arrives.
• A large semacquire group often points at lock contention, pool exhaustion, sync.WaitGroup waits, or internal runtime parking around synchronization.
• A large IO wait group points at sockets held open, stalled backends, slow clients, or a connection-management issue.

Grouping by stack is where the signal sharpens. State alone is too coarse. Ten thousand goroutines in select means little if they are spread across expected server loops. Two hundred goroutines with the same stack into one cache fill path means far more.

When you inspect a grouped view, start with the top repeated stacks. Look for the first application frame above the standard library or runtime. That frame is often where you assign ownership. Which package owns the blocked path. Which call site fans out. Which request path lacks a timeout.

Wait duration is a prioritization tool

Raw counts tell you where the herd is. Wait durations tell you where the herd has been stuck long enough to matter.

In a goroutine dump, wait annotations often appear in headers, such as minutes in a wait state. Long waits change how you read a cluster. Fifty goroutines in IO wait for a few milliseconds during peak traffic might be ordinary. The same fifty waiting for minutes is a dependency issue, a dead peer, a timeout gap, or a drain failure.

Use wait duration to separate burst effects from structural faults.

• Short waits with high counts often match normal concurrency spikes.
• Long waits with stable repeated stacks often match leaks, deadlocks, blocked shutdown hooks, or missing cancellation.
• Mixed wait times inside one stack often match a throughput bottleneck where new goroutines keep joining an existing queue.

This is also where bad instrumentation habits show up. If your code starts goroutines without clear ownership and without context propagation, dump wait times tend to expose it. You will see old goroutines parked in request-derived paths long after the request should have ended. That is a design smell. It means work escaped its lifetime boundary.

To verify what a long wait means, cross-check the top repeated stack with the code path. Does it accept a context.Context. Does it pass deadlines into network clients. Does it defer unlocks and closes along error paths. Does it spawn retries in separate goroutines. You are not proving root cause from the dump alone. You are ranking suspects from runtime evidence.

Read repeated stacks as a concurrency design review

A stack held by far more goroutines than expected is often the best clue in the whole dump. Repetition means the runtime sees many copies of the same control flow shape. That usually comes from one of four patterns.

First, unbounded fan-out. A request path launches one goroutine per item, recipient, shard, or downstream call. Under slow dependencies, the number in flight grows faster than completion.

Second, missing backpressure. You have a queue, channel, or worker pool, but admission never slows producers. Blocked goroutines pile up at the edge of the pool.

Third, contention on a shared resource. A mutex, connection pool, rate limiter, or cache fill path serializes work more than intended.

Fourth, forgotten cancellation. The work should stop when a request times out or a deploy drains traffic, but child goroutines keep waiting on reads, writes, timers, or channels.

You can verify each pattern from the stack shape.

• Unbounded fan-out often shows similar worker entry frames repeated under one request or batch path.
• Missing backpressure often shows callers blocked on chan send, semaphore acquisition, or pool checkout.
• Shared-resource contention often shows semacquire, mutex paths in sync, or identical stack traces around a cache or map guard.
• Forgotten cancellation often shows network or channel waits under request-scoped handlers long after expected lifetimes.

Where teams go wrong is over-reading one stack frame. The hot frame is not always the owner of the fault. A goroutine stuck in a database call might reflect a network partition, saturated pool, or a caller that never set deadlines. The grouped dump narrows the fault domain. It does not assign blame. Treat it as a map of pressure points.

Verify claims by reproducing structure

A useful analyzer should help you check its output against the raw dump. Grouping must be explainable. If two goroutines land in the same stack bucket, you should see why. If one bucket dominates, you should be able to inspect repeated frames and confirm the count from the pasted data.

A practical verification flow is simple.

1. Capture a SIGQUIT dump from the live process or a reproducer.
2. Paste the dump and inspect the largest groups by state.
3. Within those states, inspect the largest repeated stacks.
4. Pick the first application frame in the dominant bucket.
5. Read the code around it for cancellation, lock scope, queue bounds, and timeout behavior.
6. Compare the wait durations across the bucket. Long-lived waits deserve first attention.
7. Reproduce under load, then compare whether the same bucket grows again.

This matters because dump analysis often fails through confirmation bias. An engineer expects database trouble and reads every IO wait as proof. Another expects a lock and fixates on semacquire. Grouping and counts impose discipline. You start from the dominant structure in the runtime snapshot, then move into code.

There is another common failure mode. Teams capture one dump and stop. A single snapshot is useful, but two or three snapshots spaced apart add motion. If the same stack group grows between captures, you are looking at accumulation. If counts rotate across groups, you are looking at load shape or scheduler churn. The analyzer helps with the first-pass structure. Your job is to compare over time.

Common misreads in goroutine dumps

Some patterns look alarming and are not. Others look harmless and are costly.

A large number of goroutines is not, by itself, evidence of a bug. Many Go services keep parked goroutines for listeners, idle pools, and long-lived background loops. What matters is concentration, age, and ownership.

Runtime and standard library frames are also easy to misread. A bucket rooted in internal/poll, net, or runtime needs the first application frame above it. Without that frame, you know the mechanism of waiting, not the owner.

Another common mistake is treating all repeated stacks as equal. A repeated stack in a bounded worker pool might be expected. A repeated stack in a path that should run once per request is more suspicious. Expected concurrency has a design envelope. Compare the group count to your own architecture, not to an abstract ideal.

The last trap is ignoring shutdown behavior. Goroutine leaks often surface during deploys and drains, not peak traffic. Handlers return, but background work waits forever on channels or socket reads. In dumps, these show up as old waits in paths tied to request processing, streaming, or subscription management. If your service is hard to stop cleanly, the dump often tells you where lifecycle control broke down.

What to watch next

The strongest signal after a first dump is repeatability. Capture another dump after a short interval and check whether the same grouped stacks persist or grow. Then inspect the owning code for bounded concurrency, deadlines, and cancellation paths.

If the dominant groups involve network waits, compare them with your client timeout settings and pool limits. If they involve channels or semaphores, inspect admission control and worker ownership. The goal is simple. Turn a wall of stacks into a short list of design checks, then verify each one in code and under load.