Yapi secondary development

  1. Yapi secondary development project development environment construction database configuration
  2. Yapi Mock principle and source code parsing
  3. Socket pull and push service sorting (secondary development)
  4. Sorting out Socket packet capture Service (Secondary development)
  5. Potholes encountered and solutions

Setting up secondary development environment

Environmental requirements:

  1. Node version >= 7.6.0
  2. NPM version >= 5.0
  3. Ykit old version 0.x
  4. Make sure the mongodb database is installed locally (MINE is V4.0.26)
  5. Only MAC or Linux OS development is supported. Windows OS development is not supported
Secondary development
Install yapi
  1. Creating a project directory
The mkdir yapi && CD yapi git clone https://github.com/YMFE/yapi.git vendors - the depth = 1 # or download the zip package decompression to vendors directoryCopy the code
  1. Modify the configuration
Cp vendors/config_example.json./config.json # Modify the configuration vi./config.json after replication is completeCopy the code

The configuration is as follows: Configure the MongoDB database and Admin account.

{
  "port": "3000"."adminAccount": "[email protected]"."timeout":120000."db": {
    "servername": "127.0.0.1"."DATABASE": "yapi"."port": 27017."authSource": ""
  },
  "mail": {
    "enable": true."host": "smtp.163.com"."port": 465."from": "***@163.com"."auth": {
      "user": "***@163.com"."pass": "* * * * *"}}}Copy the code
  1. Install dependencies
CD vendors NPM install - registry https://registry.npm.taobao.org # rely on installationCopy the code
  1. Initialize the
NPM run install-server # The installation program will initialize the database index and the administrator account. The administrator account name can be configured in config.jsonCopy the code
  1. Start the project
npm run dev
Copy the code
Installing the mongodb Database

Homebrew is recommended for macOS installation

Brew tap mongodb/brew brew install [email protected]Copy the code

The following directories will be created in your MAC location after installation. Intel is different from M1

! [image-20220304143826885](/Users/calvin/Library/Application Support/typora-user-images/image-20220304143826885.png) There are two ways to run the mongodb service:

  1. Start mongodb as a macOS service using BREW
Brew Services start [email protected] brew Services stop [email protected]Copy the code
  1. Run mongodb as a background process
Mongod -- config/usr/local/etc/mongod. Conf # Intel processor mongod -- - fork config/opt/homebrew/etc/mongod. Conf # - fork M1 processor db.shutdownServer({timeoutSecs: 60}); # service shutdownCopy the code

In addition, you can add mongodb to environment variables. My terminal is ZSH

Vim ~ /. ZSHRC export PATH = "$PATH: / usr/local/Cellar/[email protected] 4.0.26 / bin" source ~ /. ZSHRCCopy the code

** Note: The root directory of macOS Catalina is read-only for security purposes. The default location for storing data in mongodb is /data/db, but you can use mongod –dbpath to specify other locations **

Sudo mkdir -p/System/Volumes/Data/Data/db # create the directory sudo chown -r id - UN/System/Volumes/Data/Data/db # to access mongod -- dbpath = / System/Volumes/Data/Data/db # start the serviceCopy the code

Yapi mock principle

Js library and jSON-schema-faker library run in sandbox mode to produce random data of various data types.

  • Mock. js is a data simulation generator. The mock. Js official
  • Json-schema-faker is a JSON-schema + fake data generator that generates fake data
  • Sandbox fixed the remote command execution vulnerability with Safeity instead of the VM library in node.js. Safeify enables Node applications to execute untrusted user-defined code in secure isolation.
Safeity principle

Execute a piece of code dynamically, eval, Function constructor, because eval and current code scope is consistent, can modify the current scope variable, there are security issues. Functions created by the Function constructor do not create closures of the current environment. They are always created in the global environment, so they can only access global variables and their own local variables at run time. Safeity uses a combination of proxy and Function constructor to execute the code found in SandobX to achieve the purpose of “escape prevention”, making it safer to execute untrusted user-defined code.

function evalute(code,sandbox) {
  sandbox = sandbox || Object.create(null);
  const fn = new Function('sandbox'.`with(sandbox){return (${code})} `);
  const proxy = new Proxy(sandbox, {
    has(target, key) {
      // Make dynamically executing code think the property already exists
      return true; }});return fn(proxy);
}
evalute('1 + 2') / / 3
evalute('console.log(1)') // Cannot read property 'log' of undefined
Copy the code
The sandbox mechanism

Node.js provides a built-in module VM that allows you to compile and run code in the V8 virtual machine context. JavaScript code can be compiled and run immediately, or compiled, saved, and run later.

const vm = require('vm');
const script = new vm.Script('m + n');
const sandbox = { m: 1.n: 2 };
const context = new vm.createContext(sandbox);
script.runInContext(context);
Copy the code

But this module is not secure, which is where the YAPI code vulnerability comes in. Executing the following code in an advanced mock prior to the yAPI bug fix will access the server’s permissions, resulting in sandbox escape and security issues.

const sandbox = this
const ObjectConstructor = this.constructor
const FunctionConstructor = ObjectConstructor.constructor
const myfun = FunctionConstructor('return process')
const process = myfun()
mockJson = process.mainModule.require("child_process").execSync("whoami && ps -ef").toString()
Copy the code

Safeity takes a further step on vm2 and manages sandbox process through process pool.

Safeity features:

  • Create a dedicated process pool for dynamic code to be executed, separate from the host application and executed in a separate process
  • You can set the maximum number of processes in a sandbox process pool
  • Support for limiting the maximum execution time for synchronous code as well as for limiting the execution time for asynchronous code
  • Support for limiting the overall CPU resource quota of the sandbox process pool (decimal)
  • Support for limiting the maximum memory limit (in m) for the entire sandbox process pool

Sandbox code used in yAPI

const Safeify = require('safeify').default;

module.exports = async function sandboxFn(context, script) {
    // Create safeify instance
    const safeVm = new Safeify({
        timeout: 3000.asyncTimeout: 60000
    })
    // fix error: TypeError: Cannot read property 'delay' of undefined
    // script += `\n return {mockJson, resHeader, httpCode, delay}`; 
    script += "; return this;";
    // Execute dynamic code
    const result = await safeVm.run(script, context)
    // Release resources
    safeVm.destroy()
    return result
}

Copy the code

Comb socket pull and push services

Background:

Driven by the engineering project, the company is committed to the decoupling of front-end and back-end work in addition to HTTP interface and socket interface. Yapi only provides mock function of HTTP interface, and mock supporting B/C socket interface and interface document management need to be developed again

Function points:
  1. Socket pull and push interface document management

  2. Socket The mock function of the pull interface

  3. Mock socket push interface and single and timed push (push at a certain frequency)

Technology stack:

React Redux front-end koa Mongoose back-end

Sequence diagram

Sorting out socket packet capture services

Background:

Packet capture tools such as Fiddle and Chales cannot capture socket interfaces. In order to facilitate testers to locate the problem quickly, the socket interface packet capture function is developed

Function points:
  1. Traffic list and details display
  2. Supports regular and custom content, RequestMsgType/TopicId, ResponseMsgType/msgType filters
  3. Sends a heartbeat to the server every 30 seconds
  4. The Websocket connection channel is closed automatically when the listening page is moved out for 10 minutes
  5. Manage the packet capture service
  6. A key to remove
  7. Break line reconnection
  8. You can manually pause and re-establish the current connection
Technology stack:

React WebSocket Webworker Back-end Koa Mongoose

Potholes encountered and solutions

  1. The socket pushes tasks by frequency
  • The introduction of node-schedule scheduled task library cannot well solve the requirement of pushing tasks at a certain frequency, because it can only be specific to seconds.

  • Process with timer, save in Map (use this one)

  • You can start a task but cannot close the current task immediately.

Cause: After a lot of checks, I thought there was an error in the interface code. Finally, I considered that two instances were opened when yAPI service was deployed, resulting in instance1 being entered when opened and Instance2 being entered when closed. However, the two instances were not shared and were independent of each other. Solution: Start only one instance running service.

  1. The socket caught
  • Long list data processing (virtual list), when tens of thousands of data are inserted into the DOM at a time, because browser rendering of the DOM tree (redraw, backflow) is very expensive, rendering is very slow, and the page can even crash.

How the virtual list works: When scrolling to load data, only the data in the visible area is loaded. And then we started the React – Virtualized tank.

  • If the Mode on the REACT Virtualized Table is virtualized, the mode on the defaultCellDatagetter.js file will be triggered if the MODE on the REACT Virtualized Table is virtualized

  • Although virtual list can solve the problem of rendering a large amount of data at one time, packet capture is pushed in real time through Websoket. Frequent DOM operation in a short period of time causes excessive CPU load, resulting in page gridlock and crash, resulting in performance problems.

Solution: Refer to the web-workers in Any-Proxy, an open source project of Ali, to enable multi-threading.

import RecordWorker from "worker-loader? inline! ./CaptureWorker.js";
const myRecordWorker = new RecordWorker(window.URL.createObjectURL(new Blob([RecordWorker.toString()])));
Copy the code
  • A performance bug that has bothered me for a long time is the setTimeout abuse problem. When sending websocket heartbeat and establishing WS channel, the heartbeat function will be called every time a data is sent. Because in the underlying CPU scheduling mechanism of the computer, timer realizes the timing function through a priority queue and an Event Loop thread. However, after a large number of events are inserted into the queue, they cannot be released even if the CPU cannot process them, making the page stuck.

Solution: Execute a timer (setInterval) after the WS is established to do heartbeat packets.