Writing in the front

Where there is life, there is reconstruction

This is not a purely technical article, but a summary of what is the back end of refactoring

Almost A month and A half of the time before and after National Day, your own website from the database, to the back end, and then to A client and reconstruct A whole C end, National Day 7 day curtilage at home yards code no problem, but for now I finish or achieve the desired, although has not changed much tall, but somehow project engineering some than before, the refactoring process is long, But I do have some experience of my own. The refactoring experience will be covered in three articles — backend Services, Nuxt applications, and Docker integration

Blog address: jooger.me

Warehouse address: backend, C end, A end

There are approximately 200 commits, welcome star, welcome message ๐Ÿ˜„. Without further ado, let’s take a look at the refactoring experience on the back end

Why refactor the back end?

There are several reasons

  1. Just want to check out the legendary enterprise framework -Egg

I would like to play it if I have never used it before. Maybe I will come to Nest later.

  1. The log system is not perfect

In the past, I thought the log was not important and there was no log backup, so I couldn’t verify the cause of many online failures. I could only say that I was too young before

  1. The deployment process is not ideal

Now docker+ Jenkins, github Webhook and Ali mirror containers are used to automate the deployment

  1. Bad code (still bad)

There is nothing to say about this, the logic layer and the controller layer are mixed together, the reuse is poor, refactoring is a matter of time

As for why TypeScript isn’t used, I’ll just say that I originally used TS and searched a few articles, but it got really uncomfortable and I gave up, but the other two projects were refactoring in TS

What was refactored?

The framework

Used to be “general operation -Koa, with some plugins, still good

Reconstruction after use is ali open an Egg, the document is really good, although I have no fully see complete documentation (advanced that part slightly under the eyes), especially the multi-process model and interprocess communication section really detailed, and illustrated an Egg is introduced in the practice of multi-process architecture, for me this contact Node pm2 directly, People who have not been exposed to Cluster are helpful.

In addition, I also recommend the Nest. Js framework, which is based on Express. I only looked at it briefly and found that it is very similar to Srping, so I may use it for reconstruction in the future

The database

I have been using mongodb for database and Mongoose for driver. This reconstruction mainly reconstructed the setting table and added notification and STAT tables

Setting table mainly stores the configuration of the website, divided into four parts

  • siteSome configurations of the C terminal
  • personalPersonal information
  • keysParameters of some third-party plug-ins, such as Ali Cloud OSS, Github, Ali Node platform (this will be talked about later), some configuration of personal email
  • Limit List interface paging, maximum number of spam comment data configuration

As for the keys, the server startup, before some service initialization parameters are often configuration in the integration tools, my side will be migrated to stored in the database, the server from the database before you start loading the configuration parameters, and then start the service, so if there are changes in parameters, also don’t have to restart the server, You only need to restart the corresponding service

Notification table mainly stores some operation notifications of C-terminal and internal system services, which currently includes 4 categories and 18 sub-categories of notification type (#L188)

The stat table collects statistics on the C-side and displays them on the A-side. Keywords, likes, and user creation generate statistics. Currently, only six operations #L217 are counted

You can see the effect

The business logic layer is separated from the Controller layer

Take a look at the Controller flow diagram before refactoring

There are three problems with doing all the business logic in the Controller and calling the Model interface directly from the logic

  1. Logic bloat, if logic is complex, oneControllerThere’s a lot of code and it’s not maintainable
  2. Each callModelAll layers need to be caught. There is no unified treatment, so it is very troublesome to modify
  3. ControllerBusiness logic reuse between

Any one of these three problems should be taken seriously

Then take a look at the refactored flowchart

After such logical separation, the above three problems are well solved

  1. Controller is very clean, the logic has been broken down, the process step by step, very clear
  2. It can be seen that a Proxy layer is added on top of the Model layer to unify the output interface for the business logic layer to call, and also to do unified catch processing here
  3. After the business logic layer is removed, all controllers can be called and reuse problems can be solved

The entire process can be combined with Egg’s Logger to quickly locate problems

As for Proxy, THIS is how I implement it

// service/proxy.js
const { Service } = require('egg')

// The Proxy needs to inherit from EggService because the other module Services need to inherit from Proxy
module.exports = class ProxyService extends Service {
    getList (query = {}) {
        return this.model.find(query, / /...).
    }
    / /... A unified interface for some models
}

// service/user.js
const ProxyService = require('./proxy')

// Inherit Proxy to define the model to which the current module belongs
module.exports = class UserService extends ProxyService {
    get model () {
        return this.app.model.User
    }
    
    getListWithComments () {}
    // Other business logic methods
}

// controller/user.js
const { Controller } = require('egg')

module.exports = class UserController extends Controller {
    async list () {
        const data = await this.service.user.getListWithComments()
        data
            ? ctx.success(data, 'User list obtained successfully')
            : ctx.fail('Failed to get user list')}}Copy the code

Logging system

As mentioned above, there is no such thing as logging before refactoring, and it can be tricky to locate and reproduce some online problems, which is a big reason WHY I’m bullish on Egg.

Egg logs have the following features

  • Log classification, classification, it has four log types (appLogger.coreLogger.errorLogger.agentLogger), 5 log levels (NONE.DEBUG.INFO.WARN.ERROR), and the print level can be configured based on environment variables
  • Unified error log,ERRORLevel logs are printed in the common-error. Log file for easy tracing
  • Log cutting, this is very nice, can be cut by day, hour, file size, generatedexample-app-web.log.YYYY-MM-DDForm log file
  • Custom logging, I didn’t use it, but if you can customize it, it’s very scalable and flexible
  • High performance. The website explains that regular logs are generated at a high frequency of Web access, and each time a log is printed, disk I/O is performed, whereas Egg uses thisLogs are written to the memory synchronously and flushed asynchronously at intervals (1 second by default)This strategy can improve performance

Deployment process

I will talk about this in a future article, in conjunction with the other two projects, and give a general refactoring process for now

Jenkins -> Jenkins pull docker image -> start container -> email (QQ) notification -> Jenkins pull docker image -> Jenkins pull docker image -> Complete the deployment

Some solutions

CTX. Body wrap

I need it every time I write reponse

ctx.status = 200
ctx.body = {/ /... }
Copy the code

It was annoying, so I implemented a middleware that encapsulates reponse operations

Now define the code map in config

// config/config.default.js

module.exports = appInfo= > {
    const config = exports = {}
    
    config.codeMap = {
        '1': 'Request failed'.200: 'Request successful'.401: 'Permission verification failed'.403: 'Forbidden'.404: 'URL resource not found '.422: 'Parameter verification failed'.500: 'Server error'
        // ...}}Copy the code

The following middleware is then implemented

// app/middleware/response.js
module.exports = (opt, app) = > {
    const { codeMap } = app.config
    const successMsg = codeMap[200]
    const failMsg = codeMap[- 1]

    return async (ctx, next) => {
        ctx.success = (data = null, message = successMsg) = > {
            if (app.utils.validate.isString(data)) {
                message = data
                data = null
            }
            ctx.status = 200
            ctx.body = {
                code: 200.success: true,
                message,
                data
            }
        }
        ctx.fail = (code = - 1, message = ' ', error = null) = > {
            if (app.utils.validate.isString(code)) {
                error = message || null
                message = code
                code = - 1
            }
            const body = {
                code,
                success: false.message: message || codeMap[code] || failMsg
            }
            if (error) body.error = error
            ctx.status = code === - 1 ? 200 : code
            ctx.body = body
        }

        await next()
    }
}
Copy the code

And then you can do that in controller

// success
ctx.success() // { code: 200, success: true, message: codeMap[200] data: null }
ctx.success(any[], 'Obtaining list succeeded') // { code: 200, success: true, message: '่Žทๅ–ๅˆ—่กจๆˆๅŠŸ' data: any[] }

// fail
ctx.fail() // { code: -1, success: false, message: codeMap[-1], data: null }
ctx.fail(- 1.'Request failed'.'Error message') // {code: -1, success: false, message: 'request failed ', error:' error ', data: null}
Copy the code

Custom unified error handling

Exceptions thrown by Controll and Service, such as

  • Exception thrown when interface parameter verification failed
  • Some internal network request services failed to throw exceptions
  • Exception thrown by model query failure
  • An exception thrown by the business logic itself

Sometimes we customize the exception’s uniform interception handling, which can be adapted to our business-defined Response code using KOA’s Middleware

// app/middleware/error.js
module.exports = (opt, app) = > {
    return async (ctx, next) => {
        try {
            await next()
        } catch (err) {
            // All exceptions raise an error event on the app, and the framework logs an error
            ctx.app.emit('error', err, ctx)
            let code = err.status || 500
            
            // If code is 200, it means that the business logic actively throws an exception. Code = -1 because I agreed that the error request status is -1
            if (code === 200) code = - 1
            let message = ' '
            if (app.config.isProd) {
                // If the environment is Production, match the pre-agreed set of requested codes
                message = app.config.codeMap[code]
            } else {
                // dev environment, so long return actual error message
                message = err.message
            }
            // This will unify the reponse to the client
            ctx.fail(code, message, err.errors)
        }
    }
}
Copy the code

Initialization of parameters before server startup

As mentioned above, some of my service configuration parameters are stored in the database, so BEFORE starting the service, I need to query the configuration parameters in the database and then start the corresponding service. Fortunately, Egg provides a self-start method to solve this problem

// app.js
module.exports = app= > {
    app.beforeStart(async() = > {const ctx = app.createAnonymousContext()
        const setting = await ctx.service.setting.getData()
        // Then you can start some services, such as mail service, anti-spam comment service, etc
        ctx.service.mailer.start()
    })
}
Copy the code

Well, everything was going well until I came across egg-Alinode, which starts in the Agent, of course, because it just reports Node Runtime system parameters to the platform. Therefore, these dirty and tiring tasks are handed over to the agent, without the management of the main process and each worker

So I needed to start the Alinode service “asynchronously”. The egg-Alinode service was started after the main process started and the fork Agent process was initialized, so it didn’t support the way I started it, so I forked the egg-Alinode repository and tweaked it a bit. We can take a look at egg-alinode-async. On the basis of supporting the original function, we use egg IPC to inform agent to initialize the Alinode service

So the code for app.js becomes the following

module.exports = app= > {
    app.beforeStart(async() = > {const ctx = app.createAnonymousContext()
        const setting = await ctx.service.setting.getData()
        / /... Start some services
        // Start alinode asynchronously in production
        if (app.config.isProd) {
            // Use IPC to send alinode event to agent to start service asynchronously
            app.messenger.sendToAgent('alinode-run', setting.keys.alinode)
        }
    })
}
Copy the code

This solved all of my parameter initialization problems

Docker and Docker-compose are added

This third article, “Website Refactoring -Docker+Jenkins Integration,” will go into more detail

Vscode debugging an egg

Check out VSCode debug Egg Perfect – Evolution history

Deficiency in

  • The test case is not perfect (although the test case is important, I really don’t want to write)
  • – TS is not used (lol)
  • The log has not been fully persisted at present, and I want to package the log and upload it to Ali Cloud for storage in the future
  • .

conclusion

After writing so much, looking back, I found that there were quite a lot of refactorings, from the reasons for the refactorings to the final results, so far so good. Moreover, the company’s project also needs to be restructured recently. I have also read some relevant articles, hoping that this can be used when writing the experience of restructuring, and also hope that the above solutions will be slightly helpful to others who have the same question. Finally, let’s talk about some of my thoughts on the relevant articles that I have taken time to read

Reconstruction requires to clarify why and when first, then talk about how and what, and finally review. Now why and when have become gradually clear, it is imperative. While how refers to the quantitative indicators, scheme design and specifications, and subsequent maintenance planning, etc. given by the business in terms of technology, what involves the realization of specific system technology. In fact, after overall planning, the complexity of reconstruction is no less than that of a brand new product, and we must pay attention to the non-technical problems in reconstruction. If it is purely technical reconstruction, we need to carefully examine why and when

Well, sauce!

Refer to the article

  • Eggjs + Docker best practices
  • VSCode debug Egg Perfect version – Evolution history
  • juejin.cn/post/1
  • . Many others are missing

Site reconstruction – background services