How can Incremental builds of Webpack multi-portal applications be up to 97% faster

Conclusion first

After the optimization of “incremental compilation” scheme, the fastest construction was selected, which was compared with the worst data of full construction, which was equivalent to speeding up construction

(125-2.81) / 125 = 97.75%

Ps: This graph was cut off because the build time was the shortest in my maintenance project. At that time, I modified a copy on HTML, a part of the code on JS, and it is true that the entry page is relatively simple

background

A project I maintained before, based on Express + Webpack + Vue, multi-page Node related front-end project server side rendering part, using Node to request the interface, and using EJS template front-end rendering part, using VUE to request the interface in the front-end and then rendering

At that time, the publishing tool used by the technical department was Walle (click on the relevant document address)

In order to meet the business requirements: 1. Need a set of automatic on-line process; 2. Ensure the online publishing and rollback time

At that time, I designed the following scheme:

The reason for designing two Git repositories

When a problem occurs online, the hope is that the code can be rolled back in less than a minute. If there is only one source repository, the build process takes time and is personally unacceptable if there is a problem online.

What the front-end build does

Perform the git pull
Webpack production environment build
Upload JS, CSS, img to CDN
Copy the node code
Commit “Online Repository”

Tips: Before “online code repository”, only node-related JS files and ejS files are included. Developers do not need to care about this repository, only in the walle publishing platform to publish online access, in the previous company, the operation of online is QA personnel

Existing problems

In the whole scheme, the “front-end construction” stage is the bottleneck, and the specific problems are as follows:

Webpack is a multi-page application, with 154 entries at the time, and as the entries grow over time, the build gets slower and slower
A small change, say a piece of text, a section of style, a link to an image, but the entire project needs to be built
Project is node and front-end code together, partial update is only need to modify the node side of the code, the theory is no front-end build
When multi-page entries are added, the js file names generated by common Mainfest and some entries with the same code will change
With so many entries, no matter how you optimize it, it will only optimize from “slow” to “slow”

Tips:

Definition of entry: the specified folder level, including both HTML files and JS files is the Webpack entry

Multiple entries: in addition to the pages required by the business, the entry also contains the previous historical activity pages. Although the later optimization uses a single page and implements the activities by associating different configurations with ID, the historical activity does not want to touch the code

Construction time: By removing the image compression function, removing the DLL public library packaging, enabling multi-process mode (using Thread-Loader to enable two processes, four-core Ali cloud server, too many startup processes will block normal other test services), the construction time has been between 100-125s, sometimes it will be slower if the server resources are insufficient

The solution

For a long time, “incremental compilation” is a good solution, that is, only compile the changed part of the content, the specific solution is as follows:

How to achieve “Incremental Compilation”

Our projects are in git repositories, and each online release will have a unique COMMIT hash value. The hash values of the two commit files can be obtained to see what files were modified in the new version relative to the online version. Using the open source library “nodegit-kit”

How do I get file dependencies

The types of files in the project: HTML, IMG, SCSS, JS, TS, vue

The difficulties in

Complexity between file dependencies

The code introduced uses the Webpack alias and needs to be able to revert back to its own address
The introduction of JS uses a mixture of ESM and REQUIRE
The Vue file consists of three parts, and determining dependencies is more complex
1. The Template section will have image dependencies
2. Js part will have image, JS, SCSS, TS, Vue dependencies
3. The SCSS part will have the dependency of picture and SCSS

The solution

In the process of understanding the principle of dependency acquisition method of WebPack, I found the package “dependency tree”, and finally chose “Madge” which depends on the package of “dependency tree” to obtain (because it can not only obtain dependencies, but also generate file dependency graph).

But there are some problems with using Madge directly:

Unable to get THE SCSS dependency on IMG
Unable to get vue file dependencies
Unable to get HTML file dependencies

Madge’s dependency fetching of JS is most efficient, much like WebPack’s solution to non-JS files is the Loader. Loader is used to convert non-JS files into JS files. Therefore, in view of the same problem, we can first use Loader to convert non-JS files into JS files, and then use Madge to obtain the dependency of files converted into JS, that is, the dependency of this file.

How to use loader

As we all know, loader is actually a function. We just point this to the generated loaderContext object during the loader-Runner call execution, because we don’t need to actually execute loader, just need to get the returned content. Therefore, the Loader function can then be executed by pointing loader’s this to a dummy loaderContext object that will contain the parameters needed for loader execution. In order to ensure the accuracy of obtaining file dependencies, the content generated by calling loader to parse will be kept in the same place as the original file.

Use of html-loader and CSS-loader

For HTML and SCSS files, htML-loader and CSS-loader will be directly called to convert them into JS files and then obtain the dependency of this JS file.

HTML source files and processed files demo: github.com/lzkui2013/m…

Process SCSS source files and processed files in the SCSS file demo: github.com/lzkui2013/m…

Core implementation code:

// html-loader processes HTML files
const loaderDealData = HtmlLoader.bind(fakeWebpack)(sourceFile);
// ...
fs.outputFileSync(`${dir}/${fileName}.js`, outCon);
/ /... For more code omitted, see the github address below for more information on how to handle HTML files

// CSS-loader processes SCSS files
cssLoader.bind(fakeWebpack)(sourceFile);
// ...
fs.outputFileSync(`${dir}/${fileName}.js`, 
   content.replace(/new URL/g.'require'));
/ /... For more code omitted, see the github address below for more information on how to process SCSS files
Copy the code

HTML file processing method: github.com/lzkui2013/m…

SCSS file processing method: github.com/lzkui2013/m…

The use of vue – loader

With HTML and SCSS file fetching dependencies resolved, let’s look at vue files. To obtain vue dependencies, the first step is to understand how vue-loader handles vue files.

1. Convert the vue file to JS. Take a vue file in the project as an example.

import { render, staticRenderFns } 
	from "./global-dialog.vue? vue&type=template&id=1adb22c4&"
import script from "./global-dialog.vue? vue&type=script&lang=js&"
export * from "./global-dialog.vue? vue&type=script&lang=js&"
import style0 from "./global-dialog.vue? vue&type=style&index=0&lang=scss&"

/ /... Omitted, contains some execution code, hot update code, no import dependencies associated
// You can see the contents of the vue project in the Webpack folder of Chrome
Copy the code

Vue-loader implements this function code address: github.com/vuejs/vue-l…

2. After the above code is returned to Webpack, vue-loader will be called according to the above import

But because each call to vUE has a Type attribute on its address this time, the selectBlock method is called

if (incomingQuery.type) { return selectBlock(/* omit, can view the code */)}
Copy the code

Vue-loader code address: github.com/vuejs/vue-l…

3. SelectBlock assigns a file suffix to each part of the selectBlock method, and then calls loaderContext.callback, which is equivalent to calling another loader. The template section of the vue-loader is handled as follows:

// template
if (query.type === `template`) {
  // if we are receiving a query with type it can only come from a *.vue file
  // that contains that block, so the block is guaranteed to exist.
  const template = descriptor.template!
  if (appendExtension) {
    loaderContext.resourcePath += '. ' + (template.lang || 'html')
  }
  loaderContext.callback(null, template.content, template.map)
  return
}
/ /... The following is omitted
Copy the code

Vue-loader code address: github.com/vuejs/vue-l…

Vue-loader process we have basically understood, then use vue-loader to achieve we need to obtain the dependency becomes very clear

1. Split the vue file into three parts; 2. Convert the template part to JS; 3. Convert the SCSS part to JS

To implement this, you can simply add all the parameters to the path, then add the contents of the vue file, call the selectBlock method, and save the returned contents.

For part of the template code, vue – loader will directly. According to its HTML way, doesn’t meet our expectations, we directly use the vue – loader/lib/loaders/templateLoader method into js. After splitting the SCSS part of the content processing, directly call the above method to deal with SCSS, the final result will generate 4 files to obtain dependencies after integration is all the dependencies of the VUE file.

Process vue source files and processed files demo: github.com/lzkui2013/m…

How to process vue files: github.com/lzkui2013/m…

We split each file, naming is certain rules, when we use Madge to obtain the corresponding dependencies, according to the naming rules, we need to integrate all dependencies and restore the original file dependencies, and then empty all generated new files

The final implementation process of the overall process is as follows

Can see through all the above plan for the whole project in the demo file dependency graph (that’s why choose madge, can generate dependency graph) : raw.githubusercontent.com/lzkui2013/m…

Cache design

If the file has not changed in git repository, then the dependencies will not change, and there is no need to re-obtain dependencies every time. After every Git comparison file changes, you only need to update the dependencies of the file whose content has changed.

So after the first build there is a dependency tree JSON cache file that contains the following contents:

Git Commit hash from the last build
List of files that each file in the code depends on

Local cache JSON file demo: github.com/lzkui2013/m…

conclusion

This case is based on vUE multi-page application, but in fact can be applied to all multi-page, other technology stack processing theory will be simpler
For single-page application construction is more complex, the current “incremental compilation” optimization scheme has a certain reference role, specific need more in-depth understanding of how webpack in single-page construction module split, how to merge the output file rules and so on

I hope this article will help you gain something.

GitHub address: github.com/lzkui2013/m…

How can Incremental builds of Webpack multi-portal applications be up to 97% faster

Conclusion first

background

The reason for designing two Git repositories

What the front-end build does

Existing problems

The solution

How to achieve “Incremental Compilation”

How do I get file dependencies

The difficulties in

The solution

How to use loader

Use of html-loader and CSS-loader

The use of vue – loader

The final implementation process of the overall process is as follows

Cache design

conclusion

Related Posts

The React project imports routes (Basic)

Year-end review: a review of the first five years of working life

React Scheduler Scheduler