Elegant closedThere seems to be no official definition or official source for Graceful Exit, but a Bing search for Graceful Exit turns up a second result: a website dedicated to women handling divorces.
In computers, graceful shutdown is simply a way of closing a program. Well, if there’s an elegant closing, there must be an inelegant closing.
Windows gracefully closed
Take the Windows computer to switch on and off this matter, long press the power button forced shutdown, or directly power off the shutdown, this is hard shutdown, the operating system can not receive any signal directly not, how elegant!
At this time, the system, or some software has not been closed before the processing, such as you worked overtime to write 4 hours of PPT to save……
But in addition to the general crash, very few people will be forced to shut down, most people’s operation or through the power option -> shutdown operation, let the operating system itself process shutdown. For example, Windows will actively close all applications before shutting down, but many applications will capture the process closing event, leading to their own unable to shut down, resulting in the system cannot shut down properly. In Office, for example, if you don’t save before closing, a pop-up box will tell you to save, which will interfere with the normal shutdown of the operating system.
Or you are using Win10, often on their own update system of that kind, if you are in the update system power forced shutdown, boot again when there may be a surprise… Halfway through writing the update file, guess what happens?
Graceful shutdown in the network
The Internet is unreliable!
TCP’s eight-part text is believed to have been memorized, four times after the wave can be disconnected, but the four wave is also established under the premise of normal closing. If you forcibly remove the network cable or power off the power, the peer end cannot detect your disconnection in time. In this case, if the peer end continues to send packets, an error will be received.
This is not enough. The application layer also has to do another heartbeat and properly and elegantly handle disconnection, Connection Reset and other errors.
Therefore, if we are writing a network program, we must provide a shutdown mechanism to shut down the socket/server normally during the shutdown event, thus reducing the number of exceptions caused by the shutdown.
How do I listen for close events?
Various languages provide this listening mechanism for closing events, but it is used differently. With this close listener, it’s easy to implement a graceful close.
JAVA Listener Off
JAVA provides a simple listening mechanism for closing events, which can receive normal closing events, such as Ctrl+C exit signals in command line programs.
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
@Override
public void run(a) {
System.out.println("Before shutdown..."); }}));Copy the codeAfter this configuration is complete, the ShutdownHook thread will be started and executed Before normal shutdown, printing Before shutdown. Of course, if you forcibly shut down the process, for example, in Windows, Kill -9 in Linux… The gods can’t listen in
Listen off in C++
There is a similar implementation in C++ that simply registers the function in the atexit function and executes the registered fnExit function before the program closes properly.
void fnExit1 (void)
{
puts ("Exit function 1.");
}
void fnExit2 (void)
{
puts ("Exit function 2.");
}
int main (a)
{
atexit (fnExit1);
atexit (fnExit2);
puts ("Main function.");
return 0;
}
Copy the codeProblems that may be encountered during shutdown
Consider a scenario where a message consumption logic pushes a peripheral system after a transaction commits successfully. The shutdown signal was received early, but due to a large number of messages, some of them have been piled up in the memory queue, but the logic of parallel consumption processing has not been completed.
At this time, some consuming threads submit transactions and receive the Force Kill signal before pushing the peripheral system. Then there will be inconsistent data. The data of this service has been dropped into the database, but no three parties are pushed…
However, as a storage engine, it must handle this inconsistency. But if you can shut down properly and let the storage engine execute safely, the risk of such inconsistencies is greatly reduced.
The process to stop
The mechanism by which a JAVA process stops is that it exits only after all non-daemons have stopped. Can a JAVA process be shut down by simply sending a shutdown signal? Certainly not!
Threads in JAVA are non-blocking threads by default, and non-daemons do not stop JVM processes as long as they stop. So when you get the shut down signal, you have to shut down all the threads, like the thread pool…
Thread the interrupt
How do threads actively close? Sorry, this really doesn’t work (the stop method has been deprecated since JAVA 1.1). It has to wait for the thread to complete its own execution, or interrupt to be added to the soft state:
private volatile boolean stopped = false;
@Override
public void run(a) {
while(! stopped && Thread.interrupted()) {// do sth...}}public void stop(a){
stopped = true;
interrupt(a); }Copy the codeWhen a thread is in WAITTING state, the interrupt method interrupts the WAITTING state, forcing it back and throwing InterruptedException. For example, if our thread is stuck on a Socket Read operation or in some lock wait state under Object.wait/JUC, calling interrupt interrupts the wait state and throws an exception.
However, if the thread is not stuck in a WAITING state, and is created in a thread pool with no soft state, then this closing strategy is not applicable.
Thread pool closure policy
ThreadPoolExecutor provides two closing methods:
shutdown– Interrupt Idle Worker thread that waits for all tasks (threads) to complete. Since idle Worker threads will be WAITING, the interrupt method directly interrupts WAITING and stops those idle threads.shutdownNow– Interrupt Specifies all Worker threads, whether idle or not. For idle threads, just like the shutdown method, interrupt is stopped directly. For working Worker threads, it does not have to be in a WAITING state, so it is not guaranteed to shutdown.
Note: Most thread pools, or frameworks that call thread pools, use shutdown instead of shutdownNow as their default shutdown policy, so executing threads are not necessarily interrupted
But as a business thread, it must be handled**InterruptedException**. Otherwise, in the event of shutdownAll, or an interruption of the manually created thread, the business thread does not respond in time, which may cause the thread to be completely unable to close
Closing strategy for the tripartite framework
In addition to the JDK thread pool, some third-party frameworks/libraries also provide methods for normal shutdown.
- In the Netty EventLoopGroup. ShutdownGracefully/shutdown – closed thread pool resources
- Reddsion redisson. shutdown – Shuts down the connection pool and destroys all resources
- In the Apache HTTPClient CloseableHttpClient. Close, close the connection pool connection, close the Evictor threads, etc
All of the major mature frameworks give you a graceful shutdown method that ensures that after you call shutdown, it destroys the resource and closes the thread/pool it created.
In particular, a tripartite framework that involves thread creation must provide a way to shut down normally, or else the thread may fail to shut down and eventually the JVM process will fail to exit properly.
Graceful shutdown in Tomcat
Tomcat shutdown script (sh version) is designed to tell you how to close:
commands:
stop Stop Catalina, waiting up to 5 seconds for the process to end
stop n Stop Catalina, waiting up to n seconds for the process to end
stop -force Stop Catalina, wait up to 5 seconds and then use kill -KILL if still running
stop n -force Stop Catalina, wait up to n seconds and then use kill -KILL if still running
Copy the codeThe design is flexible and provides you with four direct closing methods, which you can choose from.
inforceIn this mode, the process is sent a SIGTERM Signal (kill -15), which can be caught by the JVM and executes the registered ShutdownHook thread. If the process still exists after 5 seconds, Force Kill, as shown in the following figure:
ShutdownHook threads registered with Tomcat are then executed to manually close various resources, such as Tomcat’s own connections, thread pools, and so on.
And of course, the most important step is to close all apps:
// org.apache.catalina.core.StandardContext#stopInternal
// Disable filter-filter.destroy () for all applications;
filterStop();
/ / close all applications under all the Listener - a Listener. ContextDestroyed (event);
listenerStop();
Copy the codeWith the help of these two pre-closing hooks, the application can handle the closing itself, such as the Servlet Context Listener used in the XML era:
<listener>
<listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>
Copy the codeIn this Listener, Spring itself calls the Application Context’s closing method:
public void contextDestroyed(ServletContextEvent event) {
// Close Spring Application Context
this.closeWebApplicationContext(event.getServletContext());
ContextCleanupListener.cleanupAttributes(event.getServletContext());
}
Copy the codeSpring’s elegant shutdown
After executing close on Spring ApplicationContext, Spring will destroy all beans, as long as your Bean is configured with the Destroy policy or implements the AutoCloseable interface. Spring can then call Destroy when it destroys the Bean, such as the spring-wrapped thread pool ThreadPoolTaskExecutor, which DisposableBean interface:
// ThreadPoolTaskExecutor
public void destroy(a) {
shutdown();
}
Copy the codeWhen you destroy the Bean, the thread pool will execute shutdown without requiring you to manually control the thread pool shutdown.
Note here that Spring creates and destroys beans in reverse order:
So when dealing with beans with complex dependencies, you should have the front-loading Bean loaded first and the base Bean, such as the thread pool, loaded last, which is destroyed first.
Most frameworks/libraries that need to be shut down properly integrate Spring with the Spring Bean destruction portal.
The LettuceConnectionFactory integration class implements the DisposableBean interface directly, Close it inside the destroy method
// LettuceConnectionFactory
public void destroy(a) {
this.resetConnection();
this.dispose(this.connectionProvider);
this.dispose(this.reactiveConnectionProvider);
try {
Duration quietPeriod = this.clientConfiguration.getShutdownQuietPeriod();
Duration timeout = this.clientConfiguration.getShutdownTimeout();
this.client.shutdown(quietPeriod.toMillis(), timeout.toMillis(), TimeUnit.MILLISECONDS);
} catch (Exception var4) {
if (this.log.isWarnEnabled()) {
this.log.warn((this.client ! =null ? ClassUtils.getShortName(this.client.getClass()) : "LettuceClient") + " did not shut down gracefully.", var4); }}if (this.clusterCommandExecutor ! =null) {
try {
this.clusterCommandExecutor.destroy();
} catch (Exception var3) {
this.log.warn("Cannot properly close cluster command executor", var3); }}this.destroyed = true;
}
Copy the codeOther frameworks, too, integrate with Spring to destroy resources based on Spring’s destroy mechanism.
Problems with Spring’s destruction mechanism
Here is a scenario where we create a client object for MQ consumption, let’s call it XMQConsumer. In the consuming client, there is a built-in thread pool into which messages are dropped for execution when pulled.
In the message MQ consuming code, you need the database connection pool-datasource and you need to send the HTTP request -HttpClient, both of which are hosted by Spring. The DataSource and HttpClient beans are loaded in the first order. When XMQConsumer starts, the DataSource and HttpClient beans are initialized and ready to use.
However, there is no destroy-method specified for the XMQConsumer, so when the Spring container closes, it does not close the consuming client, which continues to pull and consume messages.
When Tomcat receives the shutdown signal, Spring will destroy the beans in reverse order of loading according to the shutdown process above:
Since XMQConsumer does not specify destroy, Spring only destroys #2 and #3 beans. However, the XMQConsumer thread pool thread and the main thread are asynchronous. When the first two objects are destroyed, the consumer thread is still running and needs to operate on the database and send requests through HttpClient.
Spring Boot gracefully closes
After Spring Boot, the shutdown mechanism changed a bit. Because previously Spring projects were deployed to run in Tomcat, Tomcat started Spring.
In Spring Boot (Executeable Jar), the order is reversed, since Spring is started directly and Tomcat (Embedded) is then started in Spring. Spring is responsible for shutdownHook, and finally Spring closes Tomcat.
As shown below, this is the start/stop order of the two ways:
K8S gracefully closes
If the process is still running, the process will be killed by Force Kill. If the process is still running, the process will be killed by Force Kill.
When the container runs, it sends a Kill(TERM) signal to the main process of all containers in Pod:
terminationGracePeriodSeconds, default 30 seconds), the process in the container does not complete the shutdown logic, the process will be forcibly killed.
When K8S encounters SpringBoot(Executeable Jar)
Nothing special, K8S sends TERM signal to Spring Boot process, and then executes Spring Boot ShutdownHook
When K8S encounters Tomcat
The shutdown method is the same as that of Tomcat catalina.sh, except that the initiator of the shutdown is K8S
conclusion
So much for elegant closing, what is elegant anyway? Here are three points:
- As a framework/library, be sure to provide a normal shutdown method, manually closing the thread/thread pool, destroying connection resources, FD resources, etc
- As an application, be sure to handle InterruptedException. Do not ignore this exception or you risk the process failing to exit properly
- When closing, it is important to pay attention to the order, especially for resources of the thread pool class. Make sure that the thread pool is closed first. It is safest to leave the interrupt thread alone, wait for it to complete its execution, and then close it.
reference
- Kubernetes. IO/useful/docs/con…
- Github.com/apache/tomc…
- Whatis.techtarget.com/definition/…
- www.wikiwand.com/en/Graceful…
- Docs. Spring. IO/spring – the boot…