Graceful shutdown

By default, when a user-based service receives a SIGTERM or SIGINT signal, it swiftly closes all active connections and halts all components.

However, in certain scenarios, a service might need to shut down gracefully. This means it should inform its clients of the impending shutdown, provide them with extra time to submit any pending requests, and attempt to complete all ongoing operations. This process is known as a graceful shutdown.

Graceful shutdown is deactivated by default. It can be activated by setting the graceful_shutdown_interval parameter in the configuration of the components::ManagerControllerComponent.

static_config.yaml example:

components_manager:
    graceful_shutdown_interval: 10s

With such configurations a service will switch to graceful shutdown mode after SIGTERM or SIGINT. It corresponds to components::ServiceLifetimeStage::kGracefulShutdown service lifetime stage.

At this stage the service will:

• Start failing HTTP and gRPC health checks:
  • HTTP /ping handler will return 500 errors.
  • gRPC grpc.health.v1.Health service will return NOT_SERVING status.
• Append special headers to HTTP responses and gRPC response metadata, if enabled.
• Accept new HTTP and gRPC requests and continue processing of ongoing requests for the configured time interval (10 seconds in the example above).

Graceful shutdown headers

During a graceful shutdown, a service appends special HTTP headers and gRPC metadata to outgoing responses, provided the feature is enabled. Typically, the initial metadata is employed for gRPC. However, if a graceful shutdown is triggered after the initial metadata has already been sent, trailing metadata will be utilized instead.

Graceful shutdown headers are enabled by default. They could be configured or disabled via GRACEFUL_SHUTDOWN_HEADERS dynamic configuration.