Graceful Shutdown

systemdesign/concepts go

Graceful Shutdown in Go: Practical Patterns
Let’s Go Further, Chapter 11 - Graceful Shutdown

Graceful shutdown satisfies three minimum conditions:

1. Close the entry point by stopping new requests or messages from sources like http, pub/sub systems, etc. However outgoing connections to third party services like databases or caches active.
2. Wait for all ongoing requests to finish. If a request takes too long, respond with a graceful error.
3. Release critical resources such as database connections, file locks or network listeners.

To doing graceful shutdown, first thing is catching termination signal.

Principle

Stop accepting new work → Give current work time to finish → Respect deadlines and signals.

Signals

Signal Handling (The GNU C Library)

In unix like systems, signals are basically software interrupts.

• Signal handler: A process can register a handler (a function) for a specific signal. This function runs when that signal is received.
• Default action: If no handler is registered, the process follows the default behavior for that signal. This might mean terminating, stopping, continuing, or ignoring the process.
• Unblockable signals: Some signals, like SIGKILL (signal number 9), cannot be caught or ignored. They may terminate the process.

When your Go application starts, even before your main function runs, the Go runtime automatically registers signal handlers for many signals (SIGTERM, SIGQUIT, SIGILL, SIGTRAP, and others).

• SIGTERM (Termination): A standard and polite way to ask a process to terminate. It does not force the process to stop. Kubernetes sends this signal when it wants your application to exit before it forcibly kills it.
• SIGINT (Interrupt): Sent when the user wants to stop a process from the terminal, usually by pressing Ctrl+C.
• SIGHUP (Hang up): Originally used when a terminal disconnected. Now, it is often repurposed to signal an application to reload its configuration. (Less used today)

Signal	Description	Keyboard shortcut	Catchable
SIGINT	Interrupt from keyboard	Ctrl+C	Yes
SIGQUIT	Quit from keyboard	Ctrl+\	Yes
SIGKILL	Kill process (terminate immediately)	-	No
SIGTERM	Terminate process in orderly manner	-	Yes

Timeout Awareness

It is important to know how long your application has to shut down after receiving a termination signal. For example, in Kubernetes, the default grace period is 30 seconds, unless otherwise specified using the terminationGracePeriodSeconds field. After this period, Kubernetes sends a SIGKILL to forcefully stop the application. This signal cannot be caught or handled.

Your shutdown logic must complete within this time, including processing any remaining requests and releasing resources.

Assume the default is 30 seconds. It is a good practice to reserve about 20 percent of the time as a safety margin to avoid being killed before cleanup finishes. This means aiming to finish everything within 25 seconds to avoid data loss or inconsistency.

Handling Pending Requests During Graceful Shutdown in Go

When calling server.Shutdown(ctx) in Go, it waits for either:

1. All active requests to finish cleanly, or
2. The ctx timeout to expire — after which the server stops waiting.

To avoid data loss, partial writes, or corrupted state, your handlers must respect context cancellation. This ensures they stop gracefully when the server is shutting down.

How to Notify Handlers of Shutdown:

Option A: Middleware with Cancel Channel

Wrap each request with a context that listens for shutdown:

func WithGracefulShutdown(next http.Handler, cancelCh <-chan struct{}) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx, cancel := WithCancellation(r.Context(), cancelCh)
		defer cancel()
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

Option B: Use BaseContext

Apply a global cancellable context to all connections:

ongoingCtx, cancel := context.WithCancel(context.Background())
server := &http.Server{
	Addr: ":8080",
	Handler: yourHandler,
	BaseContext: func(_ net.Listener) context.Context {
		return ongoingCtx
	},
}
// Later during shutdown:
cancel()

Avoid Ignoring Context

Replace time.Sleep() with a context-aware sleep:

func Sleep(ctx context.Context, d time.Duration) error {
	select {
	case <-time.After(d):
		return nil
	case <-ctx.Done():
		return ctx.Err()
	}
}

Important Notes:

• server.Close() forcefully shuts down the server (drops all connections immediately) — use only if Shutdown() fails or times out.
• Graceful shutdown applies not just to HTTP, but to any long-running system (like databases or message queues).
• Always prefer context-aware logic over blocking calls to ensure smooth shutdown.

Releasing Critical Resources on Shutdown

When handling shutdown in Go, avoid releasing critical resources (like DB or cache connections) immediately after receiving a termination signal—ongoing requests may still depend on them. While the OS reclaims most resources (memory, file descriptors) automatically, some components need explicit cleanup:

• Databases: Ensure connections close and transactions are committed or rolled back.
• Message queues: Flush messages and commit offsets to avoid message loss or rebalancing issues.
• External services: Manually close connections to prevent delayed cleanup from TCP timeouts.

Best practice: Shut down components in reverse order of initialization using Go’s defer, which helps maintain dependency order. Some components (like in-memory caches) may need custom shutdown routines.

Code Example

package main
 
import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"sync"
	"time"
)
 
// Dummy resource handles
type DB struct{}
type Cache struct{}
 
func connectDB() *DB {
	log.Println("Connected to DB")
	return &DB{}
}
func (db *DB) Close() {
	log.Println("DB connection closed")
}
 
func connectCache() *Cache {
	log.Println("Connected to Cache")
	return &Cache{}
}
func (c *Cache) Close() {
	log.Println("Cache connection closed")
}
 
// Global wait group for in-flight requests
var wg sync.WaitGroup
 
func main() {
	// Step 1: Setup resources in order
	db := connectDB()
	defer db.Close()
 
	cache := connectCache()
	defer cache.Close()
 
	// Step 2: Create shutdown context
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()
 
	// Step 3: Setup HTTP server
	srv := &http.Server{
		Addr: ":8080",
		Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			wg.Add(1)
			defer wg.Done()
 
			select {
			case <-time.After(2 * time.Second):
				w.Write([]byte("Request processed\n"))
			case <-ctx.Done():
				http.Error(w, "Server shutting down", http.StatusServiceUnavailable)
			}
		}),
	}
 
	// Step 4: Start server in goroutine
	go func() {
		log.Println("Server started on :8080")
		if err := srv.ListenAndServe(); err != http.ErrServerClosed {
			log.Fatalf("Listen error: %v", err)
		}
	}()
 
	// Step 5: Wait for shutdown signal
	<-ctx.Done()
	log.Println("Shutdown signal received")
 
	// Step 6: Gracefully shutdown HTTP server
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("Server shutdown failed: %v", err)
	} else {
		log.Println("HTTP server stopped")
	}
 
	// Step 7: Wait for ongoing requests to finish
	wg.Wait()
 
	log.Println("Cleanup complete. Exiting.")
}