Principle and implementation of hot deployment based on NodeJS online code

Author: The tadpole of the first prize

A hot deployment solution that allows you to make new updates work without restarting the NodeJS online service

background

As you all know, if there is any code change in the back-end service started by NodeJS, the process must be restarted for the code to take effect.

When the nodeJS process is restarted and the user accesses the service, 502 bad Gateway will appear for a short time

If you have a watch mechanism on your server

When the code on the server changes frequently, or frequently in a short period of time, it stays 502 bad Gateway

Recently, when I made relevant requirements for online service compilation, there appeared frequent changes of online service codes and frequent updates of code function modules in a short time, and the updated codes took effect when the service could not be restarted.

This involves the concept of hot deployment, where newly deployed code takes effect without restarting the service.

Let me explain the principles and implementation of hot deployment

Why code doesn't work in real time

When loading a function module via require('xx/xx.js'), Node will cache the result of require('xx/xx.js') in require.cache('xx/xx.js')

When we call require('xx/xx.js') multiple times, node does not reload and instead reads directly from the require.cache('xx/xx.js')

So when a user changes a file in xx/xx.js on the server, Node only reads the cache and does not load the user's latest code

Source code address and use

In order to realize this hot deployment mechanism, I searched the information everywhere on the Internet and stepped on a lot of holes

The following code is refined, fully operational, hot deployment fundamentals that you can extend on your own: Smart-Node-reload

Note The latest version of Node 12 May cause an error. The require.cache has been modified and the problem has been reported to the authorities

Git Clone will run directly without installation

    npm start
Copy the code

This is when hot deployment change listening is turned on

How do you see the effect

See the /hots/hot.js file

    const hot = 1
    module.exports = hot
Copy the code

Change the first line of code to const hot = 111

    const hot = 111
    module.exports = hot
Copy the code

You can see that the terminal is listening for code changes, and then dynamically loading your latest code and getting the result of execution:

Hot deployment file: hot.js, result: {'hot.js': 111 }
Copy the code

The hot deployment service listens for the code changes and reloads the code, so the partner can get the latest code execution results in real time. The whole process is running online without restarting the Node process

The source code parsing

LoadHandlers main function

const handlerMap = {};/ / cache
const hotsPath = path.join(__dirname, "hots");

// Load the file code and listen for changes in the contents of the specified folder
const loadHandlers = async() =  {// Iterate over all files in the specified folder
  const files = await new Promise((resolve, reject) =  {
    fs.readdir(hotsPath, (err, files) = {
      if (err) {
        reject(err);
      } else{ resolve(files); }}); });// Initialize all files and cache the results of each file in the handlerMap variable
  for (let f in files) {
    handlerMap[files[f]] = await loadHandler(path.join(hotsPath, files[f]));
  }

  // Listen for changes in file contents in the specified folder
  await watchHandlers();
};

Copy the code

LoadHandlers are the main function of the whole hot deployment service. We specify that the hoTS folder in the server root is used to listen for changes and hot deployment

Use fs.readdir to scan all files in the HOTS folder, loadHandler method to load and run each scanned file, and cache the result in handlerMap

File change listening is then turned on using the watchHandlers method

WatchHandlers listen for file changes

// Monitor file changes under the specified folder
const watchHandlers = async() =  {// It is recommended to use chokidar's NPM package instead of folder listening
  fs.watch(hotsPath, { recursive: true }, async (eventType, filename) = {
    // Get the absolute path to each file
    // Require. Resolve. After the path has been concatenated, it will take the initiative to determine whether the file in this path exists
    const targetFile = require.resolve(path.join(hotsPath, filename));
    // When you load a module in require, the module's data will be cached in require.cache. The next time you load the same module, it will go directly to require.cache
    // The first thing we do in a hot load deployment is clear the require.cache of the corresponding file
    const cacheModule = require.cache[targetFile];
    // Remove parent's references to the current module in require.cache, otherwise it will cause memory leaks
	/ / the record once triggered by a single line of code "attack", "https://cnodejs.org/topic/5aaba2dc19b2e3db18959e63
	/ / the line of the delete the require. Cache memory leak caused by murder, https://zhuanlan.zhihu.com/p/34702356
    if (cacheModule.parent) {
        cacheModule.parent.children.splice(cacheModule.parent.children.indexOf(cacheModule), 1);
    }
    // Clear the require.cache cache of the module corresponding to the specified path
    require.cache[targetFile] = null;

    // Reload the changed module file to achieve hot loading deployment effect and update the reloaded result to the handlerMap variable
    const code = await loadHandler(targetFile)
    handlerMap[filename] = code;
    console.log("Hot deployment file:", filename, ", execution result:, handlerMap);
  });
};

Copy the code

The watchHandlers function listens for changes to files in the specified folder, cleans and updates the cache.

Use fs.watch native function to listen for file changes under hoTS folder. When the file changes, calculate the absolute path targetFile of the file

Cache [targetFile] = null; require. Cache [targetFile] = null;

Cache [targetFile].parent is a reference to the parent level of the cache

    if (cacheModule.parent) {
        cacheModule.parent.children.splice(cacheModule.parent.children.indexOf(cacheModule), 1);
    }
Copy the code

LoadHandler Loads the file

// Load the code for the specified file
const loadHandler = filename=  {
  return new Promise((resolve, reject) =  {
    fs.readFile(filename, (err, data) = {
      if (err) {
        resolve(null);
      } else {
        try {
          // Use the Script method of the VM module to precompile the changed file code, check for syntax errors, and find out whether there are syntax errors and other errors in advance
          new vm.Script(data);
        } catch (e) {
          // Syntax error, failed to compile
          reject(e);
          return;
        }
        // After the compile passes, re-require to load the latest code
        resolve(require(filename)); }}); }); };Copy the code

The loadHandler function loads the specified file and verifies the syntax of the new file.

Read the contents of the file with fs.readfile

Use the node native VM module vm.Script method to precompile the changed file code, check syntax errors, and find out whether there are syntax errors and other errors in advance

If the check passes, use the resolve(require(filename)) method to load the file require again and automatically add the file to require.cache

The end:

That's all for hot deployment, smart-Node-reload

This code is my minimalist code, easy for everyone to read and understand, interested partners can use this code to further expand