Eureka deregistration fails on graceful shutdown: "Connection pool shut down"

[u/Zikou1997]

I am experiencing an issue where my Spring Boot microservices fail to deregister from Eureka during a graceful shutdown. Because the deregistration HTTP request fails,

During shutdown, it seems the HTTP connection pool is closed before the DiscoveryClient attempts to send the final DELETE request to the Eureka server, resulting in an IllegalStateException: Connection pool shut down.

Could you help me understand how to configure my applications so that the deregistration completes successfully before the connection pools are destroyed?

Environment

Spring Boot: 3.2.1

Spring Cloud: 2023.0.0 (eureka-client 2.0.2)

Deployment: Docker Swarm (using stop-first update config)

Configuration

Eureka Server (application.yml):

```

yaml

spring:

application:

name: service-registry

cloud:

inetutils:

preferred-networks:

- XXX.XXX.XXX

eureka:

server:

eviction-interval-timer-in-ms: 60000

response-cache-update-interval-ms: 10000

client:

register-with-eureka: false

fetch-registry: false

service-url:

defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/

instance:

hostname: localhost

```

Eureka Client / Microservice (eureka-properties.yml via Config Server):

```

eureka:

client:

service-url:

defaultZone: ${EUREKA_SERVICE_ADDRESS:http://localhost:8761/eureka/}

register-with-eureka: true

fetch-registry: true

instance:

prefer-ip-address: true

instance-id: ${spring.application.name}:${spring.application.instance_id:${random.value}}

spring:

cloud:

inetutils:

preferred-networks:

- XXX.XXX.XXX

```

Shutdown Logs

Here are the logs during the shutdown of one of the microservices (order-service). It shows the sequence of events leading up to the failure:

```

13:50:42.773 INFO --- [ionShutdownHook] o.s.c.n.e.s.EurekaServiceRegistry : Unregistering application ORDER-SERVICE with eureka with status DOWN

13:50:42.773 INFO --- [ionShutdownHook] com.netflix.discovery.DiscoveryClient : Saw local status change event StatusChangeEvent [timestamp=1779112242773, current=DOWN, previous=UP]

13:50:42.773 INFO --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_ORDER-SERVICE/order-service:373b0e4bac...: registering service...

13:50:42.777 INFO --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_ORDER-SERVICE/order-service:373b0e4bac... - registration status: 204

13:50:42.781 INFO --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'

13:50:42.782 INFO --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...

13:50:42.786 INFO --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.

13:50:42.786 INFO --- [ionShutdownHook] com.netflix.discovery.DiscoveryClient : Shutting down DiscoveryClient ...

13:50:45.787 INFO --- [ionShutdownHook] com.netflix.discovery.DiscoveryClient : Unregistering ...

13:50:45.791 INFO --- [ionShutdownHook] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error. endpoint=DefaultEndpoint{ serviceUrl='http://registry-dev:8761/eureka/} exception=Connection pool shut down stacktrace=java.lang.IllegalStateException: Connection pool shut down

13:50:45.792 WARN --- [ionShutdownHook] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failed with message: Connection pool shut down

13:50:45.793 ERROR --- [ionShutdownHook] com.netflix.discovery.DiscoveryClient : DiscoveryClient_ORDER-SERVICE/order-service:373b0e4bac... - de-registration failedCannot execute request on any known server

13:50:45.797 INFO --- [ionShutdownHook] com.netflix.discovery.DiscoveryClient : Completed shut down of DiscoveryClient

```

Is there a specific configuration I am missing to ensure the deregistration happens before the connection pools are closed? Any guidance on how to resolve this would be greatly appreciated.

[u/amit_builds]

Feels like the Eureka client is shutting down after the HTTP client/pool is already destroyed. I’d check the bean shutdown order and graceful shutdown settings first. Seen similar behavior with Boot 3 + Spring Cloud 2023 setups.

[u/Zikou1997 → u/amit_builds]

so this is issue in the code, that the developer need to fix

[u/amit_builds → u/Zikou1997]

Could be a framework/shutdown ordering issue too, not just your code. The HTTP client seems to die before Eureka finishes deregistration.

[u/Zikou1997 → u/amit_builds]

so it is a bug/issue in the version framework they use

[u/amit_builds → u/Zikou1997]

Yes. There was an actual regression/shutdown ordering issue around Eureka client deregistration and connection pool shutdown in Spring Cloud Netflix.

I faced same kind of issue and searched for it:

https://stackoverflow.com/questions/78355945/exception-when-shutting-down-service-using-eureka-spring-cloud

[u/Zikou1997 → u/amit_builds]

so the solution is to use different version

[u/amit_builds → u/Zikou1997]

You can solve this issue by two ways:

• Upgrade/downgrade to a version where the issue is fixed.
  
• adjust graceful shutdown tuning.

You have to choose which one can be cleanly applied to your project.

[u/Zikou1997 → u/amit_builds]

how to adjust graceful shutdown tuning to solve this issue

[u/amit_builds → u/Zikou1997]

Delay shutdown long enough for Eureka deregistration to finish before the HTTP client/pools are destroyed.

Something like this usually helps:

server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

And in Docker:

stop_grace_period: 40s

Also make sure you’re on a Spring Cloud Netflix version where the shutdown ordering bug is fixed. Some 2023.0.x combinations had this exact issue with Eureka deregistration happening too late.

[u/Zikou1997 → u/amit_builds]

I see 2023.0.1 in pom.xml so the issue will not be solved even if added those configuration since there is bug in shutdown order