This article documents the pitfalls and solutions of using NGINX as a reverse proxy for GRPC.
background
It is well known that Nginx is a high-performance Web server used for load balancing andThe reverse proxy. The so-called reverse proxy is corresponding to the forward proxy, which is the “proxy” in our normal sense: for example, under normal circumstances, we cannot access Google in China, if we need to access it, we need to forward it through a layer of proxy. The forward proxy represents the server (i.e., Google), while the reverse proxy represents the client (i.e., the user). When the user’s request arrives at NGINX, NGINX makes a request to the actual back-end service on behalf of the user and returns the result to the user.
(Image courtesy of Wikipedia)
Forward and reverse proxies are actually defined from the user’s point of view. Forward is the service that the proxy user requests, while reverse is the proxy user requests the service. There is an important difference between the two:
The forward proxy server is not aware of the requester, and the reverse proxy requester is not aware of the service.
Consider the example above. When you access Google via a proxy, Google can only perceive that the request is coming from a proxy server, but not directly perceive you (although it can also be traced through cookies and other means). With Nginx reverse proxying, you don’t know which back-end server the request is being forwarded to.
The most common scenario in which nginx is used for reverse proxies is what is known as the HTTP protocol. It is easy to define a reverse proxies rule by configuring the nginx.conf file:
worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; server { listen 80; server_name localhost; location / { proxy_pass http://domain; }}}Nginx has supported reverse proxies of the GRPC protocol since 1.13.10, with similar configuration:
worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; server { listen 81 http2; server_name localhost; location / { grpc_pass http://ip; }}}However, when the requirements scenarios became more complex, it became clear that NGINX’s GRPC module actually had a lot of holes, and the implementation capability was not as complete as that of HTTP, which caused problems when applying the HTTP solution
scenario
At the beginning, our scenario was very simple, using GRPC protocol to implement a simple C/S architecture:
However, this simple direct connection is not feasible in some scenarios. For example, client and server are not connected to each other in two network environments, so they cannot access the service through a simple GRPC connection. One solution is to forward it through an intermediate proxy server, using the Nginx reverse proxy GRPC method described above:
Nginx Proxy is deployed on clusters that are accessible to both environments, enabling GRPC access across network environments. The question then becomes how to configure this routing rule? Note that the target nodes of GRPC at the beginning are clear, that is, the IP addresses of server1 and server2. When a layer of Nginx Proxy is added in the middle, the objects of GRPC requests made by clients are all the IP addresses of Nginx Proxy. How does Nginx know if it needs to forward a request to server1 or server2? (Here server1 and server2 are not simply redundant deployment of the same service, and may need to decide who responds according to the properties of the request, such as user ID, etc., so it is not possible to use load balancing to randomly select a response request)
The solution
If HTTP is the protocol, there are many ways to implement it:
- Distinguish by path
/server1/service/method. Nginx forwards the request back to the original request: /server1/service/method
worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; server { listen 80; server_name localhost; location ~ ^/server1/ { proxy_pass http://domain1/; } location ~ ^/server2/ { proxy_pass http://domain2/; }}}Note the slash at the end of http://domain/. Without the slash, the path of the request would be /server1/service/method, and the server would only be able to respond to /service/method requests. This would result in a 404 error.
- Distinguished by request parameters
You can also put information about server1 in a request parameter:
worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; server { listen 80; server_name localhost; location /service/method { if ($query_string ~ x_server=(.*)) { proxy_pass http://$1; }}}}For GRPC, however, it is not so simple. For one thing, GRPC does not support URI writing, so NGINX will keep the original path of the request and cannot change the path when it is forwarded, which means that the first method above is not feasible. GRPC is based on HTTP 2.0 protocol. HTTP2 does not have the concept of QueryString. There is an item in the request header :path represents the path of the request, such as /service/method, and this path cannot carry request parameters. So :path cannot be written as /service/method? Server = server1. This means that the second method mentioned above is also not feasible.
The HTTP2 header :path specifies the path of the request, so we can modify the header :path to specify the path of the request.
worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; server { listen 80 http2; server_name localhost; location ~ ^/(.*)/service/.* { grpc_set_header :path /service/$2; grpc_pass http://$1; }}}However, the actual verification shows that this method is also not feasible. Directly modifying the PATH header will cause the server to report an error. One possible error is as follows:
rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502; transport: received the unexpected content-type "text/html"GRPC_SET_HEADER does not overwrite the result of :path, but adds a new request header, equivalent to the existence of two: paths in the request header. It may be because of this reason that the server reported 502 error.
When we think of the metadata function of GRPC at the end of our resources, we can store the information of the server in the metadata on the client side, and then forward the information of the server in the metadata to the corresponding back-end service in NGINX routing. In this way, our requirements are realized. For the Go language, setting metadata requires implementing the PerrpcCredentials interface, and passing in an instance of this implementation class when initiating the connection:
type extraMetadata struct { Ip string } func (c extraMetadata) GetRequestMetadata(ctx context.Context, uri ... string) (map[string]string, error) { return map[string]string{ "x-ip": c.Ip, }, nil } func (c extraMetadata) RequireTransportSecurity() bool { return false } func main(){ ... // NginxProxy is the IP/domain address of NginxProxy; // NginxProxy is the IP/domain address of NginxProxy; err := grpc.Dial(nginxProxy, grpc.WithInsecure(), grpc.WithPerRPCCredentials(extraMetadata{Ip: serverIp})) }Nginx is configured to forward the metadata to the corresponding server:
worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; server { listen 80 http2; server_name localhost; location ~ ^/service/.* { grpc_pass grpc://$http_x_ip:8200; }}}Note that $HTTP_X_IP is used to refer to the metadata information we pass as X-IP. This method is verified to be effective, and the client can successfully access the GRPC service of the server through Nginx Proxy.
conclusion
The documentation of the GRPC module of NGINX is too small. The official documentation only describes the purpose of a few instructions, but does not describe the metadata method. The documentation on the Internet is also very rare, so it took two or three days to troubleshoot. Summarize the whole process here, hoping to help people who encounter the same problem.