This article is from @twt community, by Bo Ya.

During this period of time, one of our application test servers experienced high CPU and failed to log in normally. The specific situation is as follows:

At about 6:00 PM on February 28th, the developer suddenly sent me a screenshot, saying that the 187 server could not log in, and asked me if I had changed the password, as shown in Figure 1:

Considering that during this period of time, only the installation of the inspection operation and maintenance monitoring tool was useful to 187 service, but it was only 10 days ago, and the password had not been changed, so I was curious to log in and have a look, and found that there was indeed a problem. It is not possible to enter the password again, as shown in Figure 2:

Go back to the source:

187 did not log in properly, but the message indicated that the server had not been shut down, but that the SSH link had been tampered with. The first thought was that the intruder was using an executable SSH backdoor and that these components were installed as services to host malware.

Out of curiosity and for just the usability of the deployment of monitoring tools, I log on operational service monitoring, found that still can collect 187 service resources such as CPU information, below three, only CPU usage on the high side, should be what is the use of the malicious software for its own services, but also explain the 187 service is available, only the new SSH connection link.

Fortunately, there was a habit of inertia before, in another computer to open a CRT, rarely to close after use. It was found that TSM service caused high CPU and high memory utilization rate of cron. I asked the developer if there was no such service, but found that it was not used, so I killed it first.

So I kill the process and change the password, but I still want to get to the bottom of the problem. When I found the process to kill, I found that the CPU immediately dropped, as shown in Figure 4:

Verify: TSM64 is a scanner responsible for spreading mining machines and backdoors through SSH brute force, which can send remote commands to download and execute malware.

After looking at the corresponding service of the process, the installation path configuration path is as follows:

root 31803 31798 84 07:44 ? 08:36:57 /tmp/.X19-unix/.rsync/c/lib/64/tsm –library-path /tmp/.X19-unix/.rsync/c/lib/64/ /usr/sbin/httpd rsync/c/tsm64 -t 505 -f 1 -s 12 -S 8 -p 0 -d 1 p ip

It is found that this service should be just a shell service, and after looking at the records of remote monitoring and collection, it is found that it was invaded at around 4:00 am on February 27th and implanted with virus, resulting in high CPU utilization rate and failure of normal login of CRT, as shown in Figure 5 and Figure 6 below:



It is the cron process that causes the high memory. Therefore, it is found through crontab-e that the process service is indeed enabled, as shown in Figure 7 below

Next, it was straightforward to stop the service, delete the files in the corresponding path and the timed jobs, and continue to observe for two days. As shown in Figure 8, it was found that there was no recurrence of the problem.



Conclusion:

Although the problem from the discovery to the solution of a total of ten minutes, but pure luck to be quickly solved, also shows that service non-functional fault processing of multidimensional, operation and maintenance technicians technical thinking divergence and comprehensive knowledge, the main or more combat is the king, the main reasons for the problem are:

One, the main reason is that the server password is too simple, resulting in the opportunity to take advantage of.

Two, the server security protection Settings are not perfect.

III. The project team members uploaded the documents without rigorous examination, which resulted in the virus in the uploaded files. This caused the problem.

Four, the server system user login authority is not perfect.

5. Don’t panic when you encounter problems. Be calm and calm.