Recently, for some reason, let me pay attention to the server again. In our server, we use Nginx as the Web server, although Nginx itself supports traffic limiting and IP limiting Settings, but in dynamic setting blacklist, nginx side itself can not do. At this point, OpenResty is ready to use.
OpenResty
The only thing to note is that since OpenResty already includes Nginx, if you already have nginx on your server, You must uninstall the current version of nginx and then install OpenResty, so you need to stop the original nginx service and back up the previous configuration file, and then install
#Delete the original nginx, remember to backup the configuration first
systemctl disable nginx.service
rm -rf /usr/lib/systemd/system/nginx.service
yum erase nginx
#Install yum - utils
yum install yum-utils
#Yum install openresty
yum install openresty
#Configure the nginx profile PATH
PATH=/usr/local/openresty/nginx/sbin:$PATH
export PATH
#Specify the configuration
nginx -c /usr/local/openresty/nginx/conf/nginx.conf
Copy the codeNow that the simple preparation for OpenResty is complete, let’s look at how to implement dynamic blacklist configuration using lua scripts.
Dynamic Blacklist script
First of all, because of the practice of using OpenResty and the following Redis components, if you are not familiar with Lua and Redis, you need to have a basic understanding of the relevant knowledge. Check out OpenResty’s description of Redis components and Nginx components.
- lua-resty-redis
- lua-nginx-module
Then, I referred to this article by space blogger [1] and made some minor adjustments.
In this process, the blogger gave me a lot of opinions and opinions, and patiently listened to my suggestions and gave answers. I would like to express my gratitude to you for giving me some other solutions in terms of operation and maintenance.
Here is the processing code, first the configuration code:
set $redis_service "127.0.0.1";
set $redis_port 6380;
set $redis_db 0;
# 1 second 50 query
set $black_count 50;
set $black_rule_unit_time 1;
set $black_ttl 3600;
set $auto_blacklist_key blackkey;
Copy the codeThere is no obvious difference between the configuration in this example. To illustrate the meanings of each configuration:
- Redis_service: indicates the IP address of the Redis server
- Redis_port: indicates the port of the redis server
- Redis_db: redis DB used
- Black_count: indicates the maximum number of access times to be blocked
- Black_rule_unit_time: specifies the TTL of kv to save the access times
- Black_ttl: specifies the lifetime of the blacklist. I did not use it because it is in permanent inventory
- Auto_blacklist_key: part key of KV
This is based on personal preferences and needs. In general, black_count and BLACK_RULE_UNIT_time will do.
Here is the specific Lua script code, most of which follows the example:
local redis_service = ngx.var.redis_service
local redis_port = tonumber(ngx.var.redis_port)
local redis_db = tonumber(ngx.var.redis_db)
local black_count = tonumber(ngx.var.black_count)
local black_rule_unit_time = tonumber(ngx.var.black_rule_unit_time)
local cache_ttl = tonumber(ngx.var.black_ttl)
local remote_ip = ngx.var.remote_addr
- count
function my_count(redis, status_key, count_key)
local key = status_key
local key_connect_count = count_key
local Status = redis:get(key)
local count = redis:get(key_connect_count)
if Status ~= ngx.null then
-- The state is connect and count is not empty and count <= number of masking times
if (Status == "Connect" and count ~= ngx.null and tonumber(count) <= black_count) then
-- Read it again
count = redis:incr(key_connect_count)
ngx.log(ngx.ERR, "count:", count)
if count ~= ngx.null then
if tonumber(count) > black_count then
redis:del(key_connect_count)
redis:set(key,"Black")
-- permanently banned
-- Redis:expire(key,cache_ttl)
else
redis:expire(key_connect_count,black_rule_unit_time)
end
end
else
ngx.log(ngx.ERR,"The visit is blocked by the blacklist because it is too frequent. Please visit later.")
return ngx.exit(ngx.HTTP_FORBIDDEN)
end
else
local count = redis:get(key)
if count == ngx.null then
redis:del(key_connect_count)
end
redis:set(key,"Connect")
redis:set(key_connect_count,1)
redis:expire(key,black_rule_unit_time)
redis:expire(key_connect_count,black_rule_unit_time)
end
end
- read the token
local token
local header = ngx.req.get_headers()["Authorization"]
if header ~= nil then
token = string.match(header, 'token (%x+)')
end
local redis_connect_timeout = 60
local redis = require "resty.redis"
local Redis = redis:new()
local auto_blacklist_key = ngx.var.auto_blacklist_key
Redis:set_timeout(redis_connect_timeout)
local RedisConnectOk,ReidsConnectErr = Redis:connect(redis_service,redis_port)
local res = Redis:auth("password");
if not RedisConnectOk then
ngx.log(ngx.ERR,"ip_blacklist connect Redis Error :". ReidsConnectErr)else
-- Connection succeeded
Redis:select(redis_db)
local key = auto_blacklist_key..":"..remote_ip
local key_connect_count = auto_blacklist_key..":key_connect_count:"..remote_ip
my_count(Redis, key, key_connect_count)
if token ~= nil then
local token_key, token_key_connect_count
token_key = auto_blacklist_key..":"..token
token_key_connect_count = auto_blacklist_key..":key_connect_count:"..token
my_count(Redis, token_key, token_key_connect_count)
end
end
Copy the codeBecause I am also a novice in lua practice, there are obvious structural problems, so I leave a hole here.
Let me explain this code first. Since I start with IP and token control, I will integrate the counts in the reference example into a function. Function originally used the set method to do the increment operation, so there would be a synchronization problem when a large number of requests came in, so I changed it slightly and used incr to do an increment operation. And at the time of entering the method to get the count value and determine whether the count size exceeds the threshold black_count, once to avoid the problem of a large number of requests.
In the following section of obtaining token, I obtained Authorization from the header according to the credential practices used in the application and extracted the token. If the token is empty, it is proved that there is no need to go through token counting processing.
Finally, it is time to connect and call the function. There is nothing left to explain here, but it is important to note the order in which function is defined and used.
Nginx conf:
server {
listen 80;
server_name blog.mintrumpet.fun;
root /~/public;
Load the configuration file
include /etc/nginx/conf.d/blacklist_params;
# specify the Lua script to execute in the request
access_by_lua_file /etc/nginx/conf.d/ip_blacklist.lua;
location/ {}error_log /etc/nginx/conf.d/log/error.log;
access_log /etc/nginx/conf.d/log/access.log;
}
Copy the codeNginx -s reload on the console, nginx -s reload can be used to dynamically add the blacklist. As for the IP and token added to the blacklist, what needs to be done in the next step is dealt with by the specific application of the server, which will not be elaborated here.
test
I used my personal server blog and apache Bench tools to do the testing.
Test an example without token to access a static file,
I set a limit of 50 times per 10 seconds, starting with 40 times for 4 concurrent accesses:
ab -n 40 -c 4 http://blog.mintrumpet.fun/dist/music.js
Copy the codeIn the execution results, you can see that 40 requests were successfully completed.
If I look at the values under Redis,
Not bad. It’s not over the limit.
Then four concurrent accesses 100 times:
ab -n 100 -c 4 http://blog.mintrumpet.fun/dist/music.js
Copy the codeAs you can see from the result, there are 49 access failures, all of which are obviously forwarded to 403.
If I look at the values under Redis,
Apparently, my IP address is blocked, and when I try to access it, I get 403 error, OK.
Let’s test an example of a token that accesses a static file with 4 concurrent accesses 100 times:
ab -n 100 -c 4 -H "Authorization:token 87BF813C6DDB9C01D4525F47908D4C9F" http://blog.mintrumpet.fun/dist/music.js
Copy the codeAs a result, the token was blocked and subsequent access was forwarded to 403.
At this point, the whole test is over, and you can see that both visitor visits and identity visits have the same effect.
summary
In fact, this approach can only be used as a general case, and in the configuration and script files, there are many areas that need to be improved, here I just have a sample, if you have a better approach and solution, please tell me in the discussion below. At the same time, when I communicated with the blogger of Sail Sky, he also told me other processing of limiting traffic and blocking to resist attacks and monitoring, such as Fail2ban and OSSEC. Nginx also provides a series of traffic limiting measures for those who are interested.
So that’s the end of OpenResty practice, check out my blog [2] blog.mintrumpet.fun/ or Reed Technologies for technical exchanges, and I’ll call it a day. Enjoy Coding!
The trumpet
Guangzhou Reed Technology Java development team
Reed Technology – Guangzhou professional Internet software service company
Seize every detail, create every beauty
Follow our official account to learn more
Want to fight with us? Lagou search for “Reed Technology” or join us by sending your resume to [email protected]
Follow us, your comments and likes support us the most
-
Sail sky, www.qhjack.cn/ ↩︎
-
Little Trumpet hut, blog.mintrumpet.fun/ ↩︎