Front-end Dependency Management and Optimization (PART 1)

One, the introduction

With the help of package management tools such as NPM and YARN, the process of front-end engineering modularization is accelerating day by day, and the dependent packages of the project are also increasing day by day. Especially if the project uses WebPack to build plug-ins that use a lot of WebPack, some development AIDS such as libraries like ESLint, PostCSS, dev-Server, and some unit tests (Jest, Mocha, enzyme) that might be needed, The node_module in the project becomes very large.

As shown in figure:

If this situation is not very desperate:

As a result, NPM install is slow and takes up a lot of space each time a team developer reinitializes the project. Furthermore, many front-end teams often use a build server on a project. Manage the code acquisition, packaging, construction and deployment of front-end projects (for example, Jenkins is used by the author’s own team to automate the hosting of front-end project packaging, construction and deployment). Project-level dependency management features such as NPM can be time consuming and resource consuming.

At the same time, the author also wants to show an attitude before the beginning of the article. The modular design of NPM itself, which makes each project independent of each other, is also the essence of its design. This article is only aimed at the interpretation of some small troubles encountered in the actual use, hoping to provide a new idea.

Add that actual YARN already has a solution – Workspaces:

Workspaces in Yarn

This article does not include YARN in the scope of discussion (the actual idea is very similar, which I discovered after writing).

This article was born under such background. Pure is the author in the project to accumulate some experience, if there is insufficient hope to point out.

Two, let’s talk about examples first

Github storage address: github.com/Roxyhuang/n…

In order to make this discussion more realistic and intuitive, the sample project has the following capabilities:

(1) provide the dev server. –

(2) Provide grammar translation (Babel)

(3) Be able to parse styles

(4) JS code style plug-in and style style check

(5) Unit test

(6) Provide style compatibility processing

Of course, the above part only reflects the level of dependence. Therefore, some dependencies may be introduced incorrectly or irrationally.

Assume that the current team uses webPack and Parcel, and React and Vue are used in the team project.

Third, the summary of the problem

Without doing any optimizations for the project and setting dependencies for the normal project, the structure should look like this:

As shown in figure:

Then we take a look at node_modules as a whole:

Case one: projects with identical dependency modules

This is a relatively common situation where there are actually multiple projects with the same dependencies

As shown in figure:

Imagine the size of the module is 162.2MB x 3 and the number of files is 1072 x 3.

It can be seen that the actual consumption of capacity is very large, and the author has made inaccurate statistics:

Module capacity and number of files required in production

Module capacity and files required for local development and testing (devDependencies) :

In fact, it is devDependencies that is responsible for the volume and quantity of our dependencies.

Of course, some libraries needed globally in production (such as React Vue) can be externalized and introduced through CDN. (that is, the reuse of CDN modules on the figure)

Case 2: Only required module dependencies in production

As shown in figure:

Module capacity and number of files required in production

Only react-redux marked in red in Dependencies in the figure,redux version and mobx react-mobx version are replaced

Module capacity and files required for local development and testing (devDependencies) :

There is no change

In this case, only the dependencies of some modules in Dependencies are different or their versions are different. React Vue may use webpack Loader, esLint plugins, etc. We will refer to this later.

In fact, it is devDependencies that is responsible for our overall dependence on such a large volume and quantity.

Case 3: Packages required for local development and testing have repeated, but no devDependencies

As shown in figure:

In this case, we also include differences in case 2 of dependencies, which also affect devDependencies.

Module capacity and number of files required in production

As shown in the figure, except for the react-Redux, the versions of Redux are different. Since the webpack + vue project appears, the dependencies of the project need to be introduced to vue or vue-Router, which are otherwise consistent.

Module capacity and files required for local development and testing (devDependencies) :

As we need to introduce vUE, we need to change devDependencies to include webPack and React plug-ins, esLint and React plug-ins, and webpack and VUE plug-ins.

In this case, only the number of files in devDependencies:

It’s still large, and there’s actually a lot of dependency overlap (17 in this example).

DevDependencies: In addition to the following dependencies:

This is the most complex situation at present, and there are several cases

(1) There are multiple front-end component solutions, such as Webpack and Parcel in this case

(2) There are differences in required modules in production, even in technology stacks

(3) The package dependencies required for local development and testing are repeated, but slightly different, with different versions

As shown in figure:

In this case, we also include the differences in case 2, and we can see that there are still many cases where there are overlapping package dependencies for local development and testing under the same component tool.

So actually if that happens, we still need to think about it and solve it.

A quick summary:

Combining the examples of case 1, case 2, case 3, and case 4, we can draw some small “rules” :

(1) devDependencies occupies a large proportion in both the volume of the module and the number of files

(2) The package in devDependencies may vary greatly from project to project, and the module in devDependencies may also differ

(3) Even if devDependencies may differ, there may still be some overlap

(4) Multiple modular schemes may lead to great differences in devDependencies of the project

In each of the first three cases of devDependencies, some of the dependencies overlap. So if there is duplication, then we need to think about reuse.

We also found that modules in multiple modular solutions (Webpack, Parcel) devDependencies differ greatly, which we will need to address later.

Ability to lock dependent modules:

It is not the focus of this article, but a brief introduction:

(1) Lock or dont’lock

This problem is an old problem discussed by the community. Locking or not locking has its supporters. However, the author does not express his attitude here, but at the same time, we hope that the optimization and “slimming” of dependency management in the future can have the ability to lock dependencies.

(2) how NPM locks dependencies and the internal loop of locking dependencies

Use semantic versioning to lock versions:

We won’t go into the details of Semantic-versioning here, but if you are interested you can take a look at the NPM documentation:

Docs.npmjs.com/about-seman…

But locking versions with semantic-versioning does not lock the internal loop of dependencies (that is, dependencies of dependencies).
NPM package – the locks to lock

Npm-package-locks we won’t go into details here, but if you’re interested, take a look:

Docs.npmjs.com/files/packa…

Our focus is that we need to rely on package-lock.json to assist in locking dependent versions as well as versions that rely on inner loops.

To sum up:

(1) devDependencies occupies a large proportion in both the volume of the module and the number of files

(2) The dependencies of projects may vary greatly from one project to another, and some of them can be externalized, so they are not within the scope of “lean management” discussed in this paper. They are mainly listed in order to be closer to the reality

(3) Although devDependencies may be different, there are still a lot of repeated and overlapping dependencies. According to the analysis of actual packages, they are the main target of “lean management”

(4) Even if there are different devDependencies, most of them are similar (as shown in Case 3), so we can extract some of the same devDependencies for “slim management”.

(5) Multiple modularization schemes may result in a big difference in devDependencie (such as webpack and Parcel in the example). Some devDependencies may vary in different versions. It is not recommended to share node_modules which are very different between the two projects.

(6) There may be version differences for the same module, so we may face the situation that an inventory has multiple versions at the same time

(7) Whether dependency locking is necessary or not, but the solution provided is expected to be available.

For shrinkwrap, use package-lock.json for shrinkwrap.

Fourth, the optimization scheme and some existing problems

The following optimizations all rely on the NodeJS module loading mechanism, so first of all, we don’t describe the complete process here, interested friends can check the official documentation or you can take a look at the simple and profound NodeJS.

How to use the NodeJS module loading mechanism

In a nutshell – to introduce a module into NodeJS, there are three steps:

Path analysis
Extension analysis
Compile implementation

We use the lookup of custom file modules (third-party NPM packages) in path analysis:

The search order of custom file modules is:

Node_modules directory in the current directory
Node_modules directory in the parent directory
You recurse up to the node_modules directory below the root directory
Recurse to the root directory

Similar to JS prototype chain search, the deeper the file path, the more time-consuming the module search, which is also the reason why it is slow.

Of course, this is by default, but in practice, we can configure NODE_PATH to recurse to the root directory, if still cannot find, give a path or multiple paths to find the specific module.

1. Configure NODE_PATH(Solution 1)

The first option actually uses the configuration NODE_PATH, which is usually configured as the global module directory where NPM i-G resides (it can actually be changed, but this example is temporarily used).

Structural changes

Case 1, case 2, and case 3 can all be converted into the following structure:

Take case 3 as an example – as shown below:

Concrete implementation:

Add the environment variable NODE_PATH to the environment variable configuration file of the corresponding system, for example, MacOS

vi /ect/profile  
/etc/bashrc / ~/.bashrc
Configuration issues, loading order, and principles are not described here
Copy the code

-> export PATH=$PATH: 
Append /usr/bin to the PATH variable

-> export NODE_PATH="/usr/lib/node_modules; /usr/local/lib/node_modules"
# specify NODE_PATH variable
Copy the code

NODE_PATH is the environment variable used in NODE to find the path provided by the module. We can use the above method to specify the NODE_PATH environment variable. And use; Split multiple different directories.

I won’t go into node’s package-loading mechanism here. The path traversal in NODE_PATH occurs by recursively searching the node_modules directory from the root of the project to the node_modules directory in the root of the file system. If the specified module has not been found, the path registered in NODE_PATH is searched.

Existing problems:

(1) Package. json will not be generated, so it is cumbersome to manage dependencies and implement incremental installation

Of course, you don’t have tonpm i -gJson, and package-lock.json, but this would only result in a linear relationship, not a tree relationship.

(2) Different versions of the same module are not supported at the same time. Therefore, there is no solution if version differences occur

Because it is a linear relationship, rather than a tree relationship, there will be a priority problem if the actual project has the same dependency version difference.

(3) Package-lock. json cannot be generated to lock dependent internal loop (dependent module dependencies)

Default:npm i -gWhen installed, package-lock.json is also not generated, so there is no locking dependency internal loop.

2. Upgrade the node_modules directory (Option 2)

In the parent directory of the project, node_modules is added to the devDependencies dependencies. That is, it can be shared between projects.

At present, the author’s ideal scheme should achieve the following objectives:

You can unify projects that build similar technology stacks and share dependencies
Maintenance and management of common modules can be achieved, and incremental installation can be achieved
The changes to the original NPM Install process will not be significant

(1) Dependent structure diagram after structural changes

All you need to do is change the project directory to the following structure

- | - below / # working directory or deployment directory    |
    |---webpack/--|---webpack-react/---|---package.json
| | |---package-lock.json
| | |---node_modules/
| | |---src/
| | |
| |--- webpack-vue/--- |---package.json
| | |---package-lock.json
| | |---node_modules/
| | |---src/
| |---node_modules/ |
| |---package.json |
| |---package.json-lock|
| | |
    |---parcel/---|--- parcel-react/---|---package.json
| | |---package-lock.json
| | |---node_modules/
| | |---src/
| | |
| |--- parcel-vue/ --- |---package.json
| | |---package-lock.json
| | |---node_modules/
| | |---src/
Copy the code

Case 1, case 2, case 3, case 4 can be transformed into the following structure:

Take case 4 as an example – as shown below:

Talk about two practices I’m exploring so far:

(2) Manually manage package.json

The next step is simple enough:

Json is a package.json file, and devDependencies is often required. NPM install is required once, and package-lock.json is generated.

These are the positions I mentioned above

|---webpack/--|---webpack-react/---|---package.json
|             |--- webpack-vue/--- |---package.json

|---parcel/---|--- parcel-react/---|---package.json
|             |--- parcel-vue/ --- |---package.json
Copy the code

In fact, you just need to maintain the normal package.json in devDependencies.

(3) using the preinstall

“The original NPM install process will not change too much”, which is actually not enough for manual management. Besides, manual management is tedious, so can we promote node_modules in devDependencies before NPM install?

The answer is: yes, is the main use of NPM preinstall in the script, of NPM script and preinstall can: docs.npmjs.com/misc/script… To get to know.

NPM’s preinstall hook actually runs before the package is installed. In fact, we can use preinstall to execute some node.js code to raise the node_modules of the overlapping module in devDependencies.

Let’s look at the steps in detail:

A. Add preinstall to the package.json project

  "scripts": {
    "preinstall": "node build/scripts/preinstall.js"."start": "webpack-dev-server --mode development --hot --progress --colors --port 3000 --open"
  }
Copy the code

Here I’m going to execute the code in the preinstall.js file in scripts under the build directory of the project

B. Switch to the parent directory of the project (i.e. the agreed technology stack directory)

Here IS just the simplest demonstration, which is to add a devDependenciesGlobal item to the project package.json:

You can actually look at the implementation code I provided

In fact, the most critical ones are:

const execPath = path.resolve('.. / ');
const preListObj = preList.devDependenciesGlobal; // The project package.json maintains its own devDependencies that overlap
const currentMd5 = md5(JSON.stringify(preListObj)); // A simple example to generate an MD5, which can be used with other mechanisms. fs.writeFileSync(`${execPath}/package.json`.JSON.stringify({"md5": currentMd5,dependencies: preListObj})); // Create a package.json file in the parent directory and write md5 and devDependenciesGlobal

let script = `cd ${execPath} && npm i`;

 // Switch to the parent directory and execute NPM install
 
exec(script, function (err, stdout, stderr) {
  console.log(stdout);
  if (err) {
   console.log('error:' + stderr);
  } else {
   console.log(stdout);
   console.log('package init success');
   console.log(`The global install in ${execPath}`); }});Copy the code

C. Changes in the actual space occupied:

(Clone examples and try them out by yourself.)

| - webpack / - | webpack - react / -- - | -- - node_modules / # 21.1 MB. 369 | | webpack - vue / -- - | -- - node_modules / # 17.8 MB, 336 | | - node_modules / # 153.3 MB. 994 | | | - parcel / - | parcel - the react / -- - | -- - node_modules / # 89.4 MB, 540 | | parcel - vue / -- - | -- - node_modules / # 71.5 MB, 491 | | - node_modules / # 21.1 MB, 118Copy the code

In this example, I will generate an MD5 based on devDependenciesGlobal when I run preinstall. If the process in Preinstall is consistent, the process in Preinstall is not executed. Of course, this is not a best practice (various subsequent practices are further described in the next article).

Of course, there are unlimited possibilities in this part, which can be improved according to your own needs (e.g. server to obtain devDependenciesGlobal and MD5).

This is just a simple demonstration that could be further improved with a few servers and Docker.

Existing problems:

(1) Windows compatibility

The author’s team encountered various problems in the Windows development environment during its use, such as:

Win10 custom preinstall process in install execution is very slow, or unable to execute
Babel or throw some errors (not specified here)

(2) The user cannot be imperceptibly

It is also the biggest disadvantage, that is, when using it, users will have a perception. When developing projects, users must take the initiative to arrange their workspace according to a specific directory structure.

(3) Local NPM version, inconsistent, may also have a greater impact on dependency

NPM script must be added to devDependencies as well.

This is just the beginning… To be continued

Using the Node module loading mechanism, we can actually improve our dependencies a lot, but we still have these problems:

(1) Windows compatibility

As mentioned above, under Windows, the author found some problems in practice. Is there any way to unify the environment?

(2) It is impossible to make users feel nothing

(3) The local NPM version has a great impact on dependencies

(5) Managing common dependencies is not standardized and automated

Therefore, in the next chapter, the author may further coordinate with this solution through other solutions

(1) The improved version of the example in this paper (remote download of package.json with public dependencies by shell, which is also used by the author’s team at present)

(2) Further improve the scheme by combining with Docker (which the author wants to further enhance)

(3) With private NPM warehouse

(4) How to standardize the management of common dependencies so that they can be standardized automatically

I’m going to dig a hole here and I’ll update that in the next post.

Conclusion and the shortcomings of this paper

After the above operations, we can carry out dependency management and division of the front-end project according to the framework. In fact, such improvement is still not the most perfect and optimization, there are still many deficiencies, looking forward to a more elegant solution to make our front-end project architecture more robust and flexible. Welcome to discuss and discuss with me. Thank you for your patience.

Front-end Dependency Management and Optimization (PART 1)

One, the introduction

Two, let’s talk about examples first

Third, the summary of the problem

Case one: projects with identical dependency modules

Case 2: Only required module dependencies in production

Case 3: Packages required for local development and testing have repeated, but no devDependencies

DevDependencies: In addition to the following dependencies:

A quick summary:

Ability to lock dependent modules:

(1) Lock or dont’lock

(2) how NPM locks dependencies and the internal loop of locking dependencies

To sum up:

Fourth, the optimization scheme and some existing problems

How to use the NodeJS module loading mechanism

1. Configure NODE_PATH(Solution 1)

Structural changes

Concrete implementation:

Existing problems:

2. Upgrade the node_modules directory (Option 2)

(1) Dependent structure diagram after structural changes

(2) Manually manage package.json

(3) using the preinstall

A. Add preinstall to the package.json project

B. Switch to the parent directory of the project (i.e. the agreed technology stack directory)

C. Changes in the actual space occupied:

Existing problems:

This is just the beginning… To be continued

Conclusion and the shortcomings of this paper

Related Posts

Webpack source code analysis – getting to know Js parsers

JavaScript proxy agent

Event sequence in JS (Event capture and bubble)