Performance tools Several common traffic replication tools

“This is the 16th day of my participation in the August More Text Challenge.

What is traffic replication?

We define the data transmission caused by users’ access to the system as traffic. Then, in the process of users’ access to the system, we can copy the incoming and outgoing data and save it for subsequent use, namely offline mode, or forward it to a new server for immediate use, namely online mode.

The application of traffic replication

In performance testing, we can use tools such as AB, WRK, Httperf, Locust, and JMeter to simulate user requests. We can also use flow “traffic replication” tools to capture production environment traffic in real time and direct it to the target test system. At the same time, these “traffic replication” tools can be used to zoom in and out of real traffic.

So some people say, well, that’s why you should just do it in real traffic, so you don’t have to worry about the business model, you have the production business model. Yes, as long as you can achieve the stress part through production flow amplification playback, you can really forget about the business scenario. But do so only if your production traffic source can cover the business scenario you want to test.

Here’s to refute the idea that some people feel that the only way to truly simulate online traffic is through production traffic playback. In fact, this view is biased.

To sum up, the advantage of the traffic replication tool is that it can copy the online traffic to the test machine, simulate the online environment in real time, and simulate the change rule of the online traffic in real time, so as to achieve the effect of bearing the online traffic in real time when the program is not online.

Common traffic replication tools

1, review

Traffic replication tools fall into the following categories:

Web server-based request replication
- Advantages: Variety of requests, cost
- Disadvantages: no versatility, loss of network delay, serious occupation of online resources
Application layer – based traffic replication tool
- Advantages: Simple implementation
- Disadvantages: But it will occupy online application resources (such as connection resources, memory resources, etc.), and may affect normal services because of high coupling degree.
Network stack – based traffic replication tool that captures packets directly from the link layer
- Advantages: less application impact
- Disadvantages: But the implementation is relatively complex

2, ngx_http_mirror_module

A plug-in introduced in Nginx 1.13.4
It is an application layer traffic replication tool

The module currently only implements two configuration instructions, which are quite simple to use:

location / {
    mirror /mirror;
    proxy_pass
}

location /mirror {
    internal;
    proxy_pass http://test_backend$request_uri;
}

Copy the code

Each mirror configuration item corresponds to a copy of the user request, so we can configure multiple mirror commands to achieve the effect of “traffic amplification”. Of course, you can also forward multiple copies to different back-end target systems.

Example configuration:

server {
    listen 8080;
    access_log /home/work/log/nginx/org.log;
    root html/org;
}

server {
    listen 8081;
    access_log /home/work/log/nginx/mir.log ; root html/mir; } upstream backend {server 127.0.0.1:8080; } upstream test_backend {server 127.0.0.1:8081; } server { listen 80; server_name localhost;# the original configuration
    location / {
        # mirror Specifies the mirror URI as /mirror
        mirror /mirror;
        Part # off | on specifies whether mirror request body (open to on, then automatically this request cache)
        mirror_request_body off;
        # specify the address of the upstream server
        proxy_pass http://backend;
    }
	
	 # mirror configuration
    location /mirror {
        # specifies that this location can only be called by "internal" requests
        internal;
        # specify the address of the upstream server
        proxy_pass http://test_backend$request_uri;
        # Set the header for mirroring traffic
        proxy_set_header X-Original-URI $request_uri; }}Copy the code

For traffic amplification, configure two mirrors:

location / {
   mirror /mirror;
   mirror /mirror;
   proxy_pass http://backend;
}
Copy the code

Nginx is very convenient to use, but online nginx usually hosts more than one service. After modifying the nginx configuration, nginx-s reload is required to make it effective. This operation should be avoided online.

3, TCPCopy

TCPCopy is a tool that copies online packets, modifies the TCP/IP header information, and sends it to the test server to cheat the TCP program of the test server. In this way, it lays a solid foundation for cheating upper-layer applications.
TCPCopy was developed in 2010 by Wang Bin from netease Technology Department based on Wang Bo’s work, and opened source in September 2011. T
TCPCopy is used together with TCPDump.
Based on C language
Address: github.com/session-rep…
Stars: 3.9 k

TCPCopy consists of two parts: TCPCopy and Intercept. TCPCopy runs on an online server and captures online requests, while Intercept runs on a secondary server and performs auxiliary work, such as passing response information to TCPCopy. The test application runs on the target server. That is to say, a secondary server is needed.

Key advantages of TCPCopy:

Protocol is not aware, can be transparent forwarding, can support any application layer protocol based on TCP, such as MySQL, Kafka, Redis and so on
Real-time forwarding, low delay
The original request IP port information can be retained and the test server can be used for statistics

At the same time, it also has the following shortcomings:

You cannot dynamically add multiple downstream servers
Due to transparent forwarding and no protocol parsing, data anomalies cannot be detected. For example, if some TCP packets are lost, the test server receives incomplete data. In addition, application layer data cannot be filtered and modified
The core components are not designed with multi-threading, so there is a bottleneck in processing capacity
Iptables needs to be modified to discard the backpackings of downstream services, which is risky to use in production or public test environments

4, GOReplay

Goreplay is an HTTP real-time traffic replication tool written in Golang. You can zoom in and out of traffic, limit traffic frequency, record requests to files for playback and analysis, and integrate with ElasticSearch to save traffic to ES for real-time analysis.
Instead of a proxy, GoReplay listens for traffic on the network interface, without changing the production infrastructure, and instead runs the GoReplay daemon on the same machine as the service.
Features: Easy to use
Address: github.com/buger/gorep…
Stars: 14.1 k

It has a simpler architecture than TCPCopy, with only one GOR component, as follows:

You just need to start a GOR process on the production server, which does all the work including listening, filtering, and forwarding. Its design follows the Unix design philosophy: everything is made up of pipes, and various inputs reuse data into outputs.

You can press the command to perform traffic replication. You don’t have to understand complicated concepts. Also supports online direct forwarding. Store it in a file for replay, N times replay.

sudo ./gor --input-raw :8000 --output-http="http://localhost:8001"
sudo ./gor --input-raw :8000 --output-file=requests.gor
Copy the code

Tcpcopy can only replicate HTTP and HTTPS traffic. Compiling is cumbersome, so you can use the compiled version directly.

Generally used in conjunction with Diffy, Diffy provides diff capability for intelligent noise reduction.

Diffy Address: github.com/twitter-arc…

5, TCPReplay

TCPReplay is a pCAP package playback tool, which can be captured using ethreal, Wireshark, or any modification of the package playback. It allows you to make arbitrary changes to packets (mainly to layer 2, 3, 4 headers), specify playback speed, etc. Tcpreplay can be used to replay captured packets to locate bugs and replay them at extremely fast speeds for stress testing.
Address: github.com/appneta/tcp…
Stars: 765

6, the JVM – the Sandbox

JVM sandbox container, a non-invasive run-time AOP solution for JVMS
Need to write code, can be suitable for some more customized scenarios
Alibaba Open Source
Address: github.com/alibaba/jvm…
Stars: 4.3 k

Overall architecture Diagram:

The sandbox can be started in two ways:

Using the JVM’s attach mechanism, attach is readily available online
To start the Java Agent, you need to add parameters on the command line. Therefore, you need to restart the Agent.

In the traffic replication scenario, attach is basically selected. The script provided is very simple to use and can be typed directly into the installation directory.

# select * from process id where process id = '2343'
./sandbox.sh -p 2343
Copy the code

The other thing is to write aOP-related code and then compile it into a JAR package and put it in your installation directory. The examples on the official website are very detailed, if you are interested, you can understand them.

7, the Sharingan

Sharingan (Chinese name: Write wheel eye) is a traffic recording and playback tool based on Golang, which is suitable for project reconstruction, regression testing and so on.
Drops open source
Address: github.com/didi/sharin…
Stars: 656

Overall architecture Diagram:

Recorder: Traffic recording module, recording traffic local file storage, sending traffic to the recording agent.
Recorder – Agent: A traffic recording agent that starts a separate process and controls the recording ratio and traffic storage.
Replayer: Traffic playback module that redirects connections to Mock Server, Mock times, adds traffic flags, and so on.
Replayer-agent: a traffic playback agent that starts a single process, queries traffic, queries and reports noise and traffic diff, plays traffic in batches, and generates coverage reports.

8 RDebug.

PHP is supported, but Java is not
Drops open source
Address: github.com/didi/rdebug…
Stars: 1.1 k

Overall architecture Diagram:

Four,

Copy request: By sending multiple copies of a request from one machine to a specified pressure measuring machine
Application scenario: The system call usage is small
Advantages: In order to make the pressure test request more close to the real service request, we try to record and replay the pressure test request from the real service traffic and use the request replication method to perform the pressure test
Disadvantages: It also faces the problem of handling dirty data from write requests. Another disadvantage is that the copied requests must intercept the response, so the machine under pressure needs to be provided separately and cannot provide normal service (can’t send the response to real users, such as sending SMS or email).

References:

[1] : Performance Test Actual Combat Lecture 30
[2] : Nginx real-time traffic replication module
[3] : Linux traffic replication tool
[4] : Traffic copy tools, which ones have you used?
[5] : Traffic replication tool
[6] : Architecture evolution of bytedance self-developed online drainage playback system
[7] : Comparison of traffic replication schemes: Tcpcopy vs Goreplay
[8] : How to simulate the real production environment? What is traffic replication? With what tools?

Performance tools Several common traffic replication tools

What is traffic replication?

The application of traffic replication

Common traffic replication tools

1, review

2, ngx_http_mirror_module

3, TCPCopy

4, GOReplay

5, TCPReplay

6, the JVM – the Sandbox

7, the Sharingan

8 RDebug.

Four,

Related Posts

Learning Elasticsearch from 0 — First introduction to Elasticsearch

Java Source code – Disruptor (Consumer)

MySQL Series Transaction Notes