Lua OpenResty Containerization (Archaeological History)

background

The company has several “ancient” projects, which are always relatively stable. However, the project always requests the peak QPS of 800K per minute at some time periods every day, which leads to some bottlenecks in the performance of the machine. At each peak period, there will always be an alarm, which is really a headache. What’s worse is that this is just one of the projects from the ancient times and it’s all deployed on physical machines, and all of them add up to close to 100 machines.

For stability (peak-shaving) and cost reasons, we ultimately decided to put all Lua OpenResty projects on the K8S cluster.

Select the appropriate OpenResty base image

By looking at the version of OpenResty in use online:

/usr/local/openresty/nginx/sbin/nginx -V nginx version: OpenResty /1.13.6.2 Built by GCC 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) OpenSSL 1.1.0H 27 Mar 2018 (Running 28 May 2019) with OpenSSL 1.1.0 k TLS SNI support enabled the configure the arguments: -- prefix = / usr/local/openresty/nginx...
Lua-V Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, Puc-Rio

OpenResty /1.13.6.2 and Lua 5.1.4 are used:

Docker pull openresty/openresty: 1.13.6.2-2 - centos

Q: Is it possible to use the smaller Alpine range?

A: Because the project relies on A number of SO libraries, all compiled by glibc, and in Alpine’s case musl-lib, incompatible.

Q: Why not recompile?

A: On the one hand, there is A risk problem. On the other hand, some SO libraries may not be found.

Find the project’s dynamic library dependencies

The Nginx configuration file

Tree - L $3 nginx/conf nginx/conf ├ ─ ─ vhosts / │ ├ ─ ─ inner. Prometheus. Nginx. Conf │ └ ─ ─ project. The nginx. Conf └ ─ ─ nginx. Conf

Self-compiled C dynamic library files, such asbinary_protocol.so

Write the dockerfile, then package the project into the container and execute:

/usr/local/openresty/nginx/sbin/nginx nginx -t

Sure enough, error report:

/usr/local/openresty/nginx/lua/init.lua:1: module 'binary_protocol' not found: no field package.preload['binary_protocol'] no file '/usr/local/openresty/nginx/lua/binary_protocol.lua' no file '/usr/local/openresty/nginx/lua_lib/binary_protocol.lua' no file '/ usr/local/openresty/nginx/luarocks/share/lua / 5.1 / binary_protocol lua' no file '/ usr/local/openresty/site/lualib binary_protocol LJBC'... ... No file '/ usr/local/openresty/nginx/luarocks/lib64 / lua / 5.1 / binary_protocol. So' no file '/usr/local/openresty/site/lualib/binary_protocol.so' no file '/usr/local/openresty/lualib/binary_protocol.so' no file '/usr/local/openresty/site/lualib/binary_protocol.so' no file '/usr/local/openresty/lualib/binary_protocol.so' no file '. / binary_protocol. So 'no file'/usr/local/lib/lua / 5.1 / binary_protocol. So 'no file '/ usr/local/openresty luajit/lib/lua / 5.1 / binary_protocol. So' no file '/ usr/local/lib/lua / 5.1 / loadall. So' no file '/ usr/local/openresty luajit/lib/lua / 5.1 / binary_protocol. So'

Q: If you look closely, you find that the SO dynamic libraries are compiled internally and provided for Lua calls. How do you find them?

A: LDD, PLDD, or using lsof to view dynamic library files.

Through the LDD, PLDD commands, you can view the dependencies associated with SO

LDD Binary_Protocol.so linude-vdso.so => (0x00007FFF40BD4000) libtolua++. So => not found ## will tell us that LDD lacks this dependency libcrypto.so.6 => not found liblog4cplus.so.2 => not found libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f458d9ef000) libm.so.6 => /lib64/libm.so.6 (0x00007f458d6ed000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f458d4d7000) libc.so.6 => /lib64/libc.so.6 (0x00007f458d10a000) /lib64/ld-linux-x86-64.so.2 (0x00007f458df1e000)

Through these methods, little by little trace, until all the dependent libraries can be found.

Luarocks external package file

From nginx.conf on the line, find the Luarocks path included in lua_package_path and lua_package_cpath. From this path, find the manifest file, which describes which Luarocks libraries are installed.

Luarocks externally depends on the installation

RUN luarocks --tree=${WORK_DIR}/luarocks install lua-cjson \ && luarocks --tree=${WORK_DIR}/luarocks install penlight \ && luarocks --tree=${WORK_DIR}/luarocks install version \ && luarocks --tree=${WORK_DIR}/luarocks install lua-resty-http  \ && luarocks --tree=${WORK_DIR}/luarocks install luaunit \ && luarocks --tree=${WORK_DIR}/luarocks install ldoc \ && luarocks --tree=${WORK_DIR}/luarocks install lua-discount \ && luarocks --tree=${WORK_DIR}/luarocks install serpent \ &&  luarocks --tree=${WORK_DIR}/luarocks install luacov \ && luarocks --tree=${WORK_DIR}/luarocks install cluacov \ && luarocks --tree=${WORK_DIR}/luarocks install mmdblua \ && luarocks --tree=${WORK_DIR}/luarocks install lua-resty-jit-uuid \ && luarocks --tree=${WORK_DIR}/luarocks install luasocket RUN luarocks --tree=/usr/local/openresty/nginx/luarocks install nginx-lua-prometheus

Problems encountered and their solutions

Problem 1: The container is OOM Killed

After analysis, it does take up a lot of memory:

The number of workers located by the ps command is very large

Solution:

Specify the number of workers: WORKER_PROCESSES 4;

Q: Why are there so many workers?

A: On k8s, the worker process started by nginx does not follow the limit we set for Pod, but is related to the node where Pod is located.

Problem 2: Nginx worker process exited on signal 9

This is because the memory quota set by Deployment is too small

Increase the Requests resource limit

resources:
  limits:
      cpu: "2000m"
      memory: "1Gi"
  requests:
      cpu: "1000m"
      memory: "512Mi"

PS: Starting 4 workers costs about 200Mi.

Attempting to index upvalue ‘result_dict’ (a nil value)

The nginx.conf is defined online, but not on the code level.

lua_shared_dict monitor_status 150m;

A tip for reducing the image size

Lend chicken unripe egg

How to access Prometheus monitoring

In OpenResty access Prometheus, https://github.com/knyar/ngin…

Install dependencies

luarocks --tree=/usr/local/openresty/nginx/luarocks install nginx-lua-prometheus

The new configuration

For nginx/conf/vhosts/project nginx. Conf increase:

lua_shared_dict prometheus_metrics 10M;
log_by_lua_block {
    metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
    metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
}

New configuration file

New nginx/conf/vhosts/inner Prometheus. Nginx. Conf

server {
    listen 8099;
    location /metrics {
        content_by_lua_block {
            metric_connections:set(ngx.var.connections_reading, {"reading"})
            metric_connections:set(ngx.var.connections_waiting, {"waiting"})
            metric_connections:set(ngx.var.connections_writing, {"writing"})
            prometheus:collect()
        }
    }
}

Update the Deployment configuration

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: ${name} namespace: ${namespace} labels: test-app: test-server spec: replicas: ${replicas} template: metadata: labels: test-app: test-server annotations: # < -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- new Prometheus. IO/scrape: "true" Prometheus. IO/path: "/ metrics" Prometheus. IO/port: "8099"

conclusion

At this point, one of Lua’s projects has been containerized. There are a lot of problems encountered along the way, and only a few major steps and problems have been documented above.