An overview,
1.1 Background
As a test, you should have encountered the following two pain points:
1. Only want to test system A under test, but need to build the desired test data layer by layer from the dependent system, which takes A long time to build, and boundary values and abnormal scenes are difficult to simulate.
2. Interface automation, UI automation, burying point automation because of service or test data instability caused by high cost of automation maintenance.
Mocks are the most common way to solve these problems. There are many excellent open source mock frameworks out there: Mockito, PowerMock, EasyMock, JMockit, etc., but these frameworks are clearly not what we want for our current business scenarios and primarily for integration testing. Because we want to support the construction of mock interface return messages to test different business scenarios without changing the development code, a non-invasive mock platform was created based on the technical architecture of external dependent services in the form of HTTP.
Hulk is a non-intrusive HTTP mock platform that supports client-side proxies, mock from the gateway layer, and mock between back-end services. Supports the configuration of the return packet function and has the permit logic. Filters will also be supported in the future, returning different mock data for different input arguments.
1.2 System Architecture
Based on Django + mitmProxy + vue + MongoDB + MySQL
The mock service is developed based on Django framework. The proxy layer is mainly developed on the basis of open source framework MitmProxy to get through the interaction with the mock system. The front-end configuration platform uses the company’s scaffolding Poizon-CLI. MongoDB and MySQL are mainly used for data storage. To improve performance, Redis will be considered in the future and the configuration information will be cached in Redis to reduce the interface response time.
1.2.1 Server mock sequence diagram
1.2.2 Mock sequence diagram on client
Mock services
2.1 Deployment and Performance
Nginx + Uwsgi + Django deployments support high concurrency.
You can build the deployment directly from the test group Jenkins.
Deployment script:
Shellcheck disable=SC2164 CD /home/dhk/workspace/hulk python37 -m venv venv Python37 -m PIP install --upgrade PIP python37 -m PIP install -r requirements.txt Install dependent libraries # shellcheck disable = SC2164 CD/home/DHK/workspace/hulk/hulk # into the uwsgi ini catalog # shellcheck disable = SC2006 # Shellcheck disable = SC2009 # get uwsgi parent pid = ` ps - ef | grep "uwsgi" | grep -v grep | awk '{print $2}' | awk 'NR==1{print}'` if [ -n "$pid" ] then uwsgi --reload uwsgi.pid else uwsgi --ini uwsgi.ini fiCopy the codePerformance: Single machine performance index on 4C8G machine
2.2 Framework design Principle
A mock service can be interpreted as an application similar to a business system that requests the address of the service’s interface and returns the corresponding message. In addition, it provides the front-end configuration function to configure customized mock interface information and host mapping of the service system corresponding to the permit interface. To design a mock service that accepts custom interface paths dynamically, you need to understand how Django handles requests and route configurations.
2.2.1 How does Django handle a request
Take a look at how the Django framework handles requests. Once you’re familiar with the principles, you can take advantage of them to design a mock service that supports custom routing. Those interested can refer to the official documentation.
When a user requests a page on a Django site, here's the algorithm the Django system uses to decide which Python code to execute: Django decides to use the root URLconf module. Normally, this is the value for the ROOT_URLCONF setting, but if the incoming HttpRequest object has the URlconf attribute (set via middleware), its value will be used instead of the ROOT_URLCONF setting. Django loads the Python module and looks for the urlpatterns available. It is a sequence of instances of Django.urls.path () and/or Django.urls.re_path (). Django iterates through each URL pattern sequentially, then stops when the requested URL matches the first pattern and matches path_info. Once there is a URL match, Djagno imports and invokes the associated view, which is a Python function (or class-based view). The view gets the following parameters: an HttpRequest instance. If the matched URL contains an unnamed group, the match from the regular expression is provided as a positional parameter. The keyword argument consists of any named part of the path expression matched and is overridden by any argument specified in the optional kwargs argument of django.urls.path() or django.urls.re_path(). Changed in Django 3.0: In older versions, keyword arguments with a value of None can also consist of named parts not provided. 5. If no URL is matched, or if an exception occurs during the match, Django calls an appropriate error-handling view. See Error Handling below. https://docs.djangoproject.com/en/3.1/Copy the code2.2.2 Route Configuration
For high-quality Web applications, the use of concise, elegant URL patterns is a very important detail. Django allows you the freedom to design your urls, regardless of the framework.Copy the codeThis uses the URL regular expression configuration rules of the Django framework to achieve an effect similar to dynamically injecting interface addresses. If an interface is requested, it will be matched in sequence until the corresponding path is matched. In addition to the interface path on the Web side, the rest will be matched with the re, such as the request: /rec/ SNS /du/ct_push_V2/recommend goes into view_mock.mock().
urlpatterns = [
path('admin/', admin.site.urls),
path('hulk/attention', view_attention.attention),
path('hulk/check', view_attention.check),
path('hulk/query_url_info', view_web.query_url_info),
path('hulk/insert_url_info', view_web.insert_url_info),
path('hulk/update_is_del_1', view_web.update_is_del_1),
path('hulk/update_url_info_doc', view_web.update_url_info_doc),
path('hulk/update_is_open', view_web.update_is_open),
path('hulk/proxy_query_url_info', view_proxy.proxy_query_url_info),
url(r'^(.*)$', view_mock.mock)
]
Copy the code2.2.3 mock logic
The mock() function will read the interface configuration information of the database first. If the interface is configured with mock, the response will be directly returned. If the interface is not configured with mock, it will continue down. To query the route configuration information of the interface, the mock system allows normal service requests, which is equivalent to layer forwarding.
Note: Only the usual POST and GET request release logic has been done. The body type of the POST request is also the default JSON.Copy the codelogger = logging.getLogger('log') def mock(request, interface): Logger.info (request.body) path = request.path # Query mongodb data = MockApiInfo().query_url_info(path) if data: Url_info = data[0] res = url_info['response'] response = JsonResponse(process_response.handle_variate(res)) Get ('response_headers'): response_headers = json.loads(url_info['response_headers']) for k, v in response_headers.items(): response.__setitem__(k, v) return response else: Headers = request. Headers if config: if request. Method == 'POST': host = request.scheme + '://' + config[0]['host'] + path headers['Content-Type'] = 'application/json' res = requests.request(request.method, url=host, headers=headers, data=request.body) logger.info(res.json()) return JsonResponse(res.json()) elif request.method == 'GET': host = request.scheme + '://' + config[0]['host'] + request.get_full_path_info() res = requests.request(request.method, url=host, headers=request.headers) logger.info(res.json()) return JsonResponse(res.json()) else: response = JsonResponse({"code": 1001, "status": 200, "msg": + path}) Response. __setitem__(" content-type ", "application/json; charset=utf-8") return responseCopy the code2.2.4 Database design
1. Configure the interface and host mapping
The relational database mysql is used to store configuration information.
Field Description:
Path -> Interface path
The host – > domain name
Bussiness -> Business domain
Description – > description
2. Interface information configuration. Considering that the interface returns json packets, it is obvious that MongoDB is suitable
Interface configuration table mock_API_info
Db. CreateCollection (” mock_api_info “).
{ "_id": ObjectId("5fcc546448bfde3202d2eaf4"), "url": "/rec/sns/du/ct_hot/recommend", "sys_name": "", "method": "", "content_type": "", "response_headers": "", "response": "{}", "rich_response": "", "description": "Recommended flow - algorithm", "is_del" : "0" and "is_open" : "1", "add_time" : "the 2020-12-06 11:47:48", "update_time" : "the 2020-12-07 11:11:40"}Copy the codeField Description:
Url -> Interface path
Sys_name -> system name
Method -> Request methods: GET,POST…
Content_type -> body Type: reserved
Response_headers -> return the header
Response -> Return the message
Rich_response -> Rich text return message: reserved field
Description – > description
Is_del -> Whether to delete: 0: not deleted, 1: deleted
Is_open -> Whether active: 0: close, 1: open
2.2.5 Front-end Configuration page
2.3 How to Mock A Server Interface
2.3.1 Configuring mock Interface Information
1. Configure the mock interface information on the mock configuration platform.
If you need to customize the header to be returned, you can configure the header information to be returned in JSON format. No configuration returns the default header.
2. After the authentication is configured, you can use an interface test tool, such as Postman, to request the configured interface. If the configured packet is returned normally, the configuration is correct.
Example: http://mock service address /rec/ SNS /du/ct_push_V2/recommend
3. Modify the server system configuration. Two configurations of the community are listed below.
2.3.2 Apollo configuration
For example, the go service of the community, the algorithm-dependent interface, and the corresponding request domain name are configured on Apollo.
To modify the interface request domain name that needs to be mock.
After saving, click Publish and restart the corresponding service.
2.3.3 Project Configuration
For example, community PHP services are in the environment.php configuration file in the project.
Change the JumpServer address directly on the server through the jumpers.
2.4 log
Connect to the root@dw-test-test-interfaces-01 server through the jumper.
Go to the /home/dhk/workshop/hulk directory.
The real-time log can be viewed at tailf uwsgi.log
All logs are saved in the logs file.
Third, the agent layer
3.1 Common Proxy Tools
App capture tools such as Charles, Fiddler, and Wireshark are often used during the test.
All the above listed tools need to be installed on their own computers, but sometimes we prefer to have a proxy server to use, such as crawler or scientific participation in e-commerce platform activities of the students will have such a tool in the tool library. There are many excellent open source frameworks, such as Whistle, that support writing JS scripts. There are anyProxy, MITmProxy and so on.
Framework selection: I chose mitmProxy as the open source framework for the simple reason that it is implemented in Python… Considering the need to do secondary development.
Comparison of MitmProxy and Charles in a scenario using the mock function
Mock with Charles:
Advantages: convenient, will not affect each other
Disadvantages: Local installation is required, mock interface is difficult to manage, and features cannot be developed again
Use the Hulk platform
Advantages: no local installation software, after configuration, as long as we connect to the proxy service can be used, more convenient support UI automation, buried point automation use. It also supports flexible open and close mocks. Support secondary development, high scalability.
Disadvantages: Share a proxy service, and you need to filter captured packets.
Note: The filter configuration of userId will be supported in the future, so that they do not affect each otherCopy the code3.2 MitmProxy after secondary development
3.2.1 introduction
At present, the proxy service has been deployed on the Intranet server. The framework is very powerful and will not be detailed here.
Introduction As the name implies, MITmProxy is a proxy for MITM, which is a man-in-the-middle attack. The agent used for man-in-the-middle attack will first forward the request like a normal agent to ensure the communication between the server and the client. Secondly, it will timely check and record the intercepted data or tamper with the data to cause the specific behavior of the server or the client. Unlike packet capture tools such as Fiddler or Wireshark, MitmProxy not only intercepts requests to help developers view and analyze them, but also implements secondary development through customized scripts. For example, Fiddler can filter out browser requests for a specific URL and view and analyze their data, but it doesn't have a high degree of customization, like: "Intercepts browser requests for this URL, empties the returned content, stores the real returned content in a database, and sends an email notification when exceptions occur." With MitmProxy, this requirement can be easily fulfilled by loading custom Python scripts. However, MITmProxy does not really launch man-in-the-middle attacks on innocent people. Since MITmProxy works at the HTTP layer, and the current popularity of HTTPS enables clients to detect and avoid man-in-the-middle attacks, mitmProxy should work properly. The client (APP or browser) must actively trust the SSL certificate of MitmProxy, or ignore the certificate exception, which means that the APP or browser belongs to the developer himself -- obviously, this is not hacking, but development or testing. In fact, the above is only the case when MITmProxy works in forward proxy mode. By adjusting the configuration, MITmProxy can also serve as a transparent proxy, reverse proxy, upstream proxy, SOCKS proxy, etc. https://www.cnblogs.com/H4ck3R-XiX/p/12624072.htmlCopy the code3.2.2 Design principle
Mock clients can intercept packets and modify response directly. When the interface requiring mock is configured, the domain name of the requested interface is changed to the domain name address of the mock service so that the client requests the mock service to achieve the same effect. This completely isolates the interaction with the business system.
Mitmproxy supports mitmproxy, mitmdump, and mitmWeb startup commands. These three commands have the same functions and can load customized scripts. The only difference is that the interface is different. In order to retain the display of front-end captured packet information, mitmWeb is selected to start, and the underlying source code is directly modified to open up the interaction with the mock service.
The modified core code snippet is as follows. All incoming requests will first determine whether a mock is needed. If a mock is needed, the request will be directly forwarded to the mock system, without the original route of the mock interface.
Hulk_api.proxy_query_url_info (path.decode(" utF-8 "))) > 0: Print ((' mock ') ' + str(path)).center(100, '=')) host = config.HULK_HOST scheme = b'http' port = 80 headers.__delitem__('Host') headers.insert(0, b'Host', bytes(host))Copy the code3.3 How to Mock Client Interfaces
Step 1: Connect agents.
Step 2: Configure the interface that requires the mock on the mock platform.
You can then mock the client interface very silky by turning off the mock button or simply disconnecting the proxy.
4. Community practice
Background: 4.60 version recommendation stream “negative feedback optimization” requirements, need to test the server in different business scenarios to return different negative feedback copywriting logic, recommendation stream interface/SNS /v2/feed/recommend rely on algorithm interface /rec/ SNS /du/ cT_hot /recommend, can not fix the content you want.
Solution: 1. Help to create data on the algorithm side (obviously expensive and troublesome) 2. Mock out the interface of the algorithm
Operation:
1. Configure the mock interface
Return message structure to fill and algorithm agreed data structure form, and change the data to their own need to test the data!
2. It is recommended to modify the CONFIGURATION of PHP Server directly on the springboard
3. Start the test
① After the mock is added, the data returned is self-configured.
(2) No mock is required. Turn off the mock switch to run the algorithm logic normally.
It is very convenient to test the logic of the service side, and also to test the value logic of the client side, and it is convenient for the acceptance of the product.
Five, the conclusion
As technology continues to evolve, there is a long way to go to create a general-purpose mock platform. For example, now popular RPC how to support? Different technical architectures have different requirements that need to be considered. Of course, all kinds of test tool platforms are designed and developed to improve efficiency and serve the business. In the future, I will continue to iterate with business characteristics, hoping to create a mock platform that is in line with business characteristics and can bring real improvements.
Article | DHK
Pay attention to the object technology, hand in hand to the cloud of technology