preface

Recently I have been making a demand:

The project has a multi-threaded execution process (involving multiple back-end interfaces) that is executed in a polling manner, requiring a reduced frequency of requests in certain circumstances. For example, the HTTP status code returns 5xx.

Because it is executed by child threads polling, which is pure data-layer logic, the situation involved is complicated. If Charles is used for mock during self-test, only fixed simulation can be carried out, and the author wants to achieve a similar effect:

The simulated back-end service has a problem, and after some time the request is normal.

So the idea was to implement an HTTP proxy like Charles. Mock several fixed interfaces in this proxy server code. This article is part of the HTTP proxy principle and implementation (1) of the reading notes, detailed analysis of the principle can be read in conjunction with this article. In addition, I studied the implementation principle of Charles. This article uses Node.js to implement the proxy server code, readers can also use other languages based on the principles described below.

Ps: This article is only for HTTP proxy. We will not discuss HTTPS for the moment. We will discuss how to handle HTTPS requests later.

Charles implementation principle

You can read the article Charles Agent Principle. Simply put, Charles is essentially HTTP hijacking. Thanks to the principle of HTTP communication, due to symmetric encryption, the process of communication between the client and the server is easy to be tampered with by the middleman, just as the middleman earns the difference 🤪.

Normal communication

graph LR
Client --request---> Server
Server --response---> Client

The role of the proxy server is the middleman, the client will first through the proxy, and then the proxy and the server communication, and finally the proxy will return the result of the communication to the client. Because HTTP is symmetrically encrypted, the communication data is equivalent to plaintext for the proxy server, so the proxy can modify the communication data according to its own Settings.

graph LR
Client --request---> Proxy --request---> Server
Server --response---> Proxy --response---> Client

So, back to the effect I want to achieve

At the time of the HTTP request, theProxyThe proxy server layerModify the response message and return it toClientThe client.
In addition, ensure that HTTPS requests are normal.

Principles of HTTP Proxy

To achieve the above mentioned effect, there are two types of agents involved: 1. 2. Tunnel agent

General agent

Ordinary proxy is the middleman profit mode mentioned above: the client sends a request packet to the proxy server, and the proxy server processes the request from the target server according to the contents of the packet, or returns the mock data to the client. This gives you the following figure and code:

graph LR
Client --request---> Proxy --request---> Server
Server --response---> Proxy --response---> Client

var http = require('http')
var net = require('net')
var url = require('url')

function request(cReq, cRes) {
    var u = url.parse(cReq.url)

    var options = {
        hostname : u.hostname, 
        port     : u.port || 80.path     : u.path,       
        method     : cReq.method,
        headers     : cReq.headers
    }

    console.log(new Date() + 'request Received request :' + cReq.url)
    
    var pReq = http.request(options, function(pRes) {
        cRes.writeHead(pRes.statusCode, pRes.headers)
        pRes.pipe(cRes)
    }).on('error'.function(e) {
        console.log("NormalRequest error :" + e)
        cRes.end()
    });

    cReq.pipe(pReq)
}

http.createServer()
    .on('request', request)
    .listen(8007.'0.0.0.0')
Copy the code

Create a service listener8007Port (Set the port number based on the actual situation), to monitorrequestEvents.
requestEvent, which parses the content of the request from the client and makes a request to the target server, sending the response data back to the client.

Tunnel proxy

Tunnel proxy in the HTTP proxy is mainly used to process HTTPS requests, that is, to ensure that HTTPS requests are not affected. It is defined as: “An HTTP client requests a tunnel agent through the CONNECT method to create a TCP connection to any destination server and port and blind forward subsequent data between the client and server.” Note that this is blind forwarding, that is, the Proxy server is not aware of the requested data content during the HTTPS request, it only serves as a delivery function.

Just like “send express”, after the seller delivers goods to the buyer, the express logistics is impossible to open your express package to check the contents, it only serves as the transportation role between the buyer and the seller.

Therefore, in the tunnel Proxy scenario, after receiving the CONNECT request, the Proxy will open a TCP connection with the target server (PS: generally, it accesses port 443 of the server). Subsequently, the client and the target server will communicate directly through this connection, and the transmission content is about asymmetric encryption of both sides. For Proxy, only the address and port number of the target server can be obtained, and the communication content cannot be decrypted.

Based on the above analysis, the code for the tunnel broker is as follows

function connect(cReq, cSock) {
    var u = url.parse('http://' + cReq.url);
    console.log("Connect received request :" + cReq.url)
    var pSock = net.connect(u.port, u.hostname, function() {
        cSock.write('the HTTP / 1.1 200 Connection Established \ r \ n \ r \ n');
        pSock.pipe(cSock);
    }).on('error'.function(e) {
        cSock.end();
    });

    cSock.pipe(pSock);
}

http.createServer()
    .on('connect', connect)
    .listen(8007.'0.0.0.0');
Copy the code

Create a service and listenconnectThe event
connectEvents are received according to the target serverAddress and port numberEstablish a socket connection and channel the socket sent by the client.

Note that the CONNECT event for this HTTP proxy will only fire on an HTTPS request. Because the proxy server detects that the request is an HTTPS and cannot process it itself, it automatically processes it into the form of CONNECT method. Here is an analysis of the timing of the trigger using OkHttp’s source code (ps: version 3.14.9 is used), requesting www.baidu.com

String url = "https://www.baidu.com";
OkHttpClient okHttpClient = new OkHttpClient.Builder().build();
Request request = new Request.Builder().url(url)
        .get()
        .build();
Call call = okHttpClient.newCall(request);
call.enqueue(new Callback() {
    @Override
    public void onFailure(Call call, IOException e) {}@Override
    public void onResponse(Call call, Response response) throws IOException {}});// RealConnection.java
private void connectSocket(int connectTimeout, int readTimeout, Call call,
    EventListener eventListener) throws IOException {
    Proxy proxy = route.proxy();
    Address address = route.address();

    rawSocket = proxy.type() == Proxy.Type.DIRECT || proxy.type() == Proxy.Type.HTTP
      ? address.socketFactory().createSocket()
      : new Socket(proxy);

    eventListener.connectStart(call, route.socketAddress(), proxy);
    rawSocket.setSoTimeout(readTimeout);
    try {
      Platform.get().connectSocket(rawSocket, route.socketAddress(), connectTimeout);
    } catch (ConnectException e) {
      ConnectException ce = new ConnectException("Failed to connect to " + route.socketAddress());
      ce.initCause(e);
      throwce; }... }Copy the code

Get (). ConnectSocket (rawSocket, route.socketAddress(), connectTimeout); The SOCKET connection is made, and the CONNECT event of the HTTP proxy responds.

To see how this works, you can read another article I wrote about OkHttp implementing TCP three-way handshake: How OkHttp manages TCP three-way handshake and four-way wave.

Mock request error

The combination of the two proxy scenarios above results in a simple HTTP proxy template. For a specified interface, you can return a specific error code. On this basis, you can add some user-defined conditions. For example, an interface error occurs within 2 minutes and the interface returns to normal after 2 minutes. There is a 60% chance of a random error.

var http = require('http')
var net = require('net')
var url = require('url')

let startTimestamp = Date.parse(new Date())
function request(cReq, cRes) {
    var u = url.parse(cReq.url)
    var options = {
        hostname : u.hostname, 
        port     : u.port || 80.path     : u.path,       
        method     : cReq.method,
        headers     : cReq.headers
    };

    console.log(new Date() + 'request Received request :' + cReq.url)

    if (cReq.url == 'http://xxx.com/xxx') {
        cRes.writeHead(503, { 
            'Access-Control-Allow-Origin': The '*'.'Access-Control-Allow-Credentials': true.'Proxy-Connection': 'keep-alive'.'Content-Length': 0.'Content-Type': 'application/json' })
        cRes.end()
    } else {
        var pReq = http.request(options, function(pRes) {
            cRes.writeHead(pRes.statusCode, pRes.headers)
            pRes.pipe(cRes)
        }).on('error'.function(e) {
            cRes.end()
        });

        cReq.pipe(pReq)
    }
}

function interceptControl() {
    if (Date.parse(new Date()) - startTimestamp < 2 * 60 * 1000) {
        return true
    } else {
        let random = Math.random()
        console.log('Intercept random number :' + random)
        return random < 0.6}}function connect(cReq, cSock) {
    var u = url.parse('http://' + cReq.url)
    console.log("Connect received request :" + cReq.url)
    var pSock = net.connect(u.port, u.hostname, function() {
        cSock.write('the HTTP / 1.1 200 Connection Established \ r \ n \ r \ n');
        pSock.pipe(cSock)
    }).on('error'.function(e) {
        cSock.end()
    });

    cSock.pipe(pSock)
}

http.createServer()
    .on('request', request)
    .on('connect', connect)
    .listen(8007.'0.0.0.0')
console.info('Proxy port 8007')    
Copy the code

In the development self-test, you can connect to the agent by setting the IP address and port number (8007) of the PC on the wifi network Settings of the mobile phone.

The last

This article is for the study of HTTP proxy notes, through the ordinary proxy and tunnel proxy implementation of a simple HTTP proxy. Used to customize some data mock scenarios while developing self-tests. Of course, Charles is still a good choice for simple interface data mocks, with Map Local, Map Remote, ReWrite, and other powerful code logic. Interested in the principle and implementation of HTTP proxy (ii), about HTTPS proxy Settings, the author has not been successful, there is time to study.

Reference article:

Principle and Implementation of HTTP Proxy (PART 1)

A brief analysis of Charles agent principle

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

For mock data, I implemented a simple HTTP proxy

preface

Charles implementation principle

Principles of HTTP Proxy

General agent

Tunnel proxy

Mock request error

The last

For mock data, I implemented a simple HTTP proxy

preface

Charles implementation principle

Principles of HTTP Proxy

General agent

Tunnel proxy

Mock request error

The last

Related Posts

Configure: error: cannot guess build type; you must specify one

How does FFMPEG push stream to server save video as a file?

Business people build their own applications so that data is not obsolete