takeaway

NPM is a package management tool that supports the thriving NodeJS community with its excellent package versioning mechanism. Understanding its internal mechanism is very helpful to deepen our understanding of module development, front-end engineering configuration to speed up our troubleshooting (I believe many students have received a variety of dependency problems) speed.

This paper analyzes the package management mechanism of NPM in detail from three aspects: package.json, version management and dependency installation.

Parse package.json

In Node.js, a module is a library or framework, as well as a Node.js project. Node.js projects follow a modular architecture. When we create a Node.js project, we create a module that must have a description file, package.json. It’s the most common configuration file we use, but have you really looked into it in detail? The configuration of a reasonable package.json file directly determines the quality of our project, so I will first take you to analyze the detailed configuration of package.json.

1.1 Mandatory Attributes

There are many properties in package.json, of which only two must be filled in: Name and Version, which form the unique identity of an NPM module.

NPM package naming rules

Name is the module name, which should be named in accordance with some official specifications and recommendations:

  • The package name becomes the module URL, a parameter in the command line, or a folder name. Any non-URL-safe characters cannot be used in the package name. Validate-npm-package-name can be used to check whether the package name is valid.

  • Semantic package names can help developers find the packages they need faster and avoid getting the wrong packages by accident.

  • If the package name contains some symbols, the symbols must be different from the existing package name

For example, because react-native already exists, neither react.native nor reactNative can be created.

  • If your package name is too close to an existing package name for you to publish it, then it is recommended to publish the package to your scope.

For example, if the user name is conard, the scope is @conard, and the published package can be @conard/react.

Check whether the package is occupied

Name is the unique identifier of a package and must not be the same as other package names. You can run the NPM view packageName command to check whether the package is occupied and view some basic information about it:

If the package name has never been used, a 404 error will be thrown:

In addition, you can go to https://www.npmjs.com/ for more detailed package information.

1.2 Description

A basic description

{
  "description": "An enterprise-class UI design language and React components implementation"."keywords": [
    "ant"."component"."components"."design"."framework"."frontend"."react"."react-component"."ui"]}Copy the code

Description Adds the description of the module for others to know about your module.

Keywords are used to add keywords to your module.

Of course, they have a very important role, is conducive to module retrieval. When you use the NPM Search module, the description and keywords are matched. Good description and keywords will help your module get more accurate exposure:

The developer

There are two fields describing developers: Author and Ficol3, where author refers to the lead author of a package, one author for one person. That leads to contributors. One leads to arrays of contributors, and the characterization of humans can go both in a string and in the following structure:

{ 
    "name" : "ConardLi"."email" : "[email protected]"."url" : "https://github.com/ConardLi"
}
Copy the code

address

{
  "homepage": "http://ant.design/"."bugs": {
    "url": "https://github.com/ant-design/ant-design/issues"
  },
  "repository": {
    "type": "git"."url": "https://github.com/ant-design/ant-design"}},Copy the code

Homepage specifies the homepage of the module.

Repository is used to specify a code repository for a module.

Bugs Specifies an address or mailbox where people who have questions about your module can go to raise them.

1.3 Dependency Configuration

Our project may depend on one or more external dependencies. Depending on their purpose, we configure them under the following properties: Dependencies, devDependencies, peerDependencies, bundledDependencies, optionalDependencies.

Configuration rules

Before introducing several dependency configurations, let’s take a look at the configuration rules for dependencies. You might see a dependency package configuration like this:

"Dependencies" : {" antd ":" ant - design/ant - design# 4.0.0 - alpha. 8 ", "axios" : "^ 1.2.0", "test - js" : "file:.. / test ", "test2 - js" : "http://cdn.com/test2-js.tar.gz", "core - js" : "^" 1.1.5,}Copy the code

Dependency configuration follows the following configuration rules:

  • Dependency package name :VERSION
    • VERSIONIs one that followsSemVerCanonical version number configuration,npm installThe package that matches the specified version range will be downloaded to the NPM server.
  • Dependent package name :DWONLOAD_URL
    • DWONLOAD_URLIt’s a downloadable onetarballZip package address, module installation will be this.tarDownload and install it locally.
  • Dependent package name :LOCAL_PATH
    • LOCAL_PATHIs a local dependency package path, for examplefile:.. /pacakges/pkgName. This works when you test one locallynpmPackage, this method should not be applied to online.
  • Dependent package name :GITHUB_URL
    • GITHUB_URL 即 github 的 username/modulenameFor example:ant-design/ant-design, you can also specify latertag 和 commit id.
  • Dependent package name :GIT_URL
    • GIT_URLThe clone code basegit url, which follows the following form:
<protocol>://[<user>[:<password>]@]<hostname>[:<port>][:][/]<path>[#<commit-ish> | #semver:<semver>]
Copy the code

Protocal can be in the following forms:

  • git://github.com/user/project.git#commit-ish
  • git+ssh://user@hostname:project.git#commit-ish
  • git+ssh://user@hostname/project.git#commit-ish
  • git+http://user@hostname/project/blah.git#commit-ish
  • git+https://user@hostname/project/blah.git#commit-ish

dependencies

Dependencies specify the module on which the project runs. Dependencies can be configured for both development and production environments, for example

 "dependencies": {
      "lodash": "^4.17.13",
      "moment": "^2.24.0",
 }
Copy the code

devDependencies

There are some packages that you might only use in your development environment, such as ESLint, which you use to check code specifications, jEST, which you use for testing. Users using your packages will work fine without installing these dependencies, but installing them will take more time and resources. So you can add these dependencies to devDependencies, which are installed and managed locally when you install NPM, but not in production:

"DevDependencies" : {" jest ":" ^ 24.3.1 ", "eslint" : "^ 6.1.0",}Copy the code

peerDependencies

PeerDependencies specifies the compatibility of the version on which the module you are developing depends and the version of the dependencies installed by the user.

The package.json configuration in Ant-Design is as follows:

"PeerDependencies" : {" react ":" > = 16.0.0 ", "the react - dom" : "> = 16.0.0"}Copy the code

When you’re developing a system that uses Ant-Design, you’ll definitely need to rely on React as well. Ant Design also relies on React. The React version is 16.0.0 for stable operation, and the React version is 15.x for development:

At this point, ant-Design uses React and introduces it:

import * as React from 'react';
import * as ReactDOM from 'react-dom';
Copy the code

You’re taking the host environment, the React version of your environment, and that can cause some problems. Specifying peerDependencies above in nPM2 would force the host environment to install versions of react@>=16.0.0 and react-dom@>=16.0.0.

Npm3 will not require the dependencies specified in peerDependencies to be forcibly installed. Instead, nPM3 will check the installation after the installation and print a warning to the user if it is not correct.

"Dependencies" : {" react ":" 15.6.0 ", "antd" : "^ 3.22.0"}Copy the code

For example, I relied on the latest version of ANTD in my project, and then on version 15.6.0 of React, and will be given the following warnings when doing a dependency installation:

optionalDependencies

In some cases, dependencies may not be strongly dependent, and they are optional. You want NPM install to continue running when the dependencies cannot be obtained. You can place the dependencies in optionalDependencies. Note that the configuration in optionalDependencies overwrites dependencies, so you only need to configure it in one place.

Of course, when referring to installed dependencies in optionalDependencies, be sure to handle exceptions, otherwise an error will be reported if the module does not get it.

bundledDependencies

Unlike the previous ones, the value of bundledDependencies is an array that specifies modules that will be packaged together when the package is published.

  "bundledDependencies": ["package1" , "package2"]
Copy the code

1.4 protocol

{
    "license": "MIT"
}
Copy the code

The license field is used to specify the open source license of the software. The open source license specifies in detail the rights that other people have after acquiring your code, which operations can be performed on your code, and which operations are prohibited. There are many variations of the same agreement. If the agreement is too loose, it will cause the author to lose many rights of the work. If the agreement is too strict, it is not convenient for users to use and spread the work.

Agreement can be divided into two categories, open source and commercial software, for commercial agreement, or legal notices and the license agreement, each software will have its own set of prose, written by the software author or specialized lawyers, for most people to take time and effort to write numerous long license agreement, choose a popular open source protocol is a good choice.

Here are some of the major open source agreements:

  • MITAs long as the user includes the copyright notice and license notice in the project copy, they can do whatever they want with your code without any liability on your part.
  • Apache: similar toMIT, it also contains provisions for contributors to provide patent licenses to users.
  • GPLUsers who modify project code must publish their changes when they redistribute source code or binary code.

For more details on open source agreements, go to choosealicense.com/.

1.5 Related to directories and Files

Program entrance

{
  "main": "lib/index.js",}Copy the code

The main property can specify the main entry file of the program, for example, the module entry lib/index.js specified by antd above, when we introduce ANTD in our code: import {notification} from ‘antd’; What is actually introduced is the module exposed in lib/index.js.

Command line tool entry

When your module is a command line tool, you need to specify an entry for the command line tool that specifies the mapping between your command name and a locally specified file. For a global installation, NPM will link the executable to /usr/local/bin using symbolic links, or to./node_modules/.bin/ for a local installation.

{
    "bin": {
    "conard": "./bin/index.js"}}Copy the code

When your package is installed globally: NPM creates a soft link in /usr/local/bin with the name conard pointing to “./bin/index.js” under the globally-installed conard package. When you execute conard on the command line, the js file will be invoked.

I won’t expand too much here, but more on this in a subsequent article on command line tools.

Publishing file Configuration

{
    "files": [
      "dist"."lib"."es"]}Copy the code

The files property describes the list of files that you publish to the NPM server. If you specify a folder, the contents of that folder will be included. We can see that the downloaded package has the following directory structure:

In addition, you can configure a.npmignore file to exclude files and prevent a large number of junk files from being pushed to NPM, the same rules as you use.gitignore. The.gitignore file can also act as a.npmignore file.

man

The man command is a help command in Linux. You can use the man command to view the help information in Linux, such as the command help, configuration file help, and programming help.

If your Node.js module is a global command line tool, you can specify the address of the document searched by the man command in package.json via the man property.

The man file must end with a number, or.gz if compressed. The numbers indicate which part of man the file will be installed into. If the man file name does not start with the module name, the installation will be prefixed with the module name.

For example, the following configuration:

{ 
  "man" : [ 
    "/Users/isaacs/dev/npm/cli/man/man1/npm-access.1"."/Users/isaacs/dev/npm/cli/man/man1/npm-audit.1"]}Copy the code

On the command line, enter man npm-audit:

Specification item List

A Node.js module is implemented based on the CommonJS modular specification. According to the CommonJS specification, the module directory must contain the following directories in addition to package.json:

  • bin: Directory for storing executable binaries
  • lib: Directory for storing JS code
  • doc: Directory for storing documents
  • test: directory for storing unit test case code
  • .

While you may not be organized or named exactly as described above in the module directory, you can specify how your directory structure corresponds to the specification structure described above by specifying the directories property in package.json. There is no other use of the Directories property.

{
  "directories": {
    "lib": "src/lib/"."bin": "src/bin/"."man": "src/man/"."doc": "src/doc/"."example": "src/example/"}}Copy the code

However, the official documentation states that while this property is not important at the moment, there may be some variations in the future, such as the markdown file in doc and the example file in Example, which may be displayed in a friendly way.

1.6 Script Configuration

script

{
  "scripts": {
    "test": "jest --config .jest.js --no-cache"."dist": "antd-tools run dist"."compile": "antd-tools run compile"."build": "npm run compile && npm run dist"}}Copy the code

Scripts are used to configure the abbreviations of some script commands. Scripts can be used in combination with each other. These scripts cover the entire project life cycle and can be invoked using NPM Run Command after configuration. If it is the NPM keyword, it can be called directly. For example, the above configuration specifies the following commands: NPM run test, NPM run dist, NPM run compile, and NPM run build.

config

The config field is used to configure environment variables used in the script, such as the following configuration, which can be obtained in the script using process.env.npm_package_config_port.

{
  "config" : { "port" : "8080"}}Copy the code

1.7 Publishing The Configuration

preferGlobal

If your Node.js module is used primarily to install to global command-line tools, set this value to true and users will be warned when installing the module locally. This configuration does not prevent the user from installing, but rather prompts the user to prevent problems caused by incorrect use.

private

If the private property is set to true, NPM will refuse to publish it, in order to prevent a private module from being unintentionally published.

publishConfig

  "publishConfig": {
    "registry": "https://registry.npmjs.org/"
  },
Copy the code

More detailed configuration when publishing modules, for example you can configure to publish only a tag, and configure the private NPM source to publish to. For more detailed configuration, see npm-config

os

If you develop a module that runs only on Darwin, you need to ensure that Windows users do not install your module to avoid unnecessary errors.

Using OS properties to help you do this, you can specify that your modules can only be installed on certain systems, or specify a blacklist of systems that cannot be installed:

"os" : [ "darwin", "linux" ]
"os" : [ "!win32" ]
Copy the code

For example, I assigned a test module to a system blacklist: “OS” : [“! Darwin “]. When I installed it on this system, I got the following error:

In the Node environment, you can use process.platform to determine the operating system.

cpu

Similar to the OS above, we can use CPU attributes to more precisely limit the user’s installation environment:

"cpu" : [ "x64", "ia32" ]
"cpu" : [ "!arm", "!mips" ]
Copy the code

In the Node environment, you can use process.arch to determine the CPU architecture.

Second, analysis package version management mechanism

Nodejs cannot succeed without NPM’s excellent dependency management system. Before introducing the entire dependency system, it is important to understand how NPM manages the version of dependencies. This chapter describes the version release specification for NPM packages, how to manage versions of various dependencies, and some best practices for package versioning.

2.1 Viewing the NPM Package Version

You can run NPM view Package Version to see the latest version of a package.

Run the NPM view conard versions command to view all released versions of a package on the NPM server.

Run the NPM ls command to view the version information of all packages in the dependency tree of the repository.

2.2 SemVer specification

Module versions in NPM packages are required to follow the SemVer specification, a guiding, unified version number representation rule drafted by Github. It’s essentially a Semantic Version.

SemVer specification official website: semver.org/

The standard version

The standard version number of the SemVer specification is in the X.Y.Z format, where X, Y, and Z are non-negative integers and zero padding before digits is prohibited. X is the major release number, Y is the minor release number, and Z is the revision number. Each element must be incremented numerically.

  • Major Version Number (major) : When you make incompatible API changes
  • Secondary Version Number (minor) : When you make a backward compatible feature addition
  • Revision number (patch) : When you make a backward-compatible problem fix.

For example, 1.9.1 -> 1.10.0 -> 1.11.0

First version

When a release is large, unstable, and may not meet the expected compatibility requirements, you may want to release an advanced release first.

The prior version number can be added to the major version number. Second version number. Revision number is followed by a join number followed by a series of identifiers separated by periods and version compilation information.

  • Internal version (alpha) :
  • Public Beta version (beta) :
  • Release candidate for official releaserc: that is,Release candiate

The version of the React

Here’s a look at the React history:

It can be seen that the version is issued in strict accordance with SemVer specification:

  • The version number is strictly followedMajor version. Minor version. Revision numberFormat name
  • The version is strictly incremental:16.8.0 -> 16.8.1 -> 16.8.2
  • Release major versions or major changes firstalpha,beta,rcSuch as prior version

The release

After modifying some functionality of an NPM package, it is often necessary to release a new version. We usually modify package.json directly to the specified version. If done wrong, it is easy to mess up the version number. We can do this with a command that conforms to Semver specification:

  • npm version patch: Indicates the upgrade version number
  • npm version minor: Indicates the upgrade version
  • npm version major: Indicates the major version

Use version 2.3 Tools

There are certainly some version numbers that need to be worked on during development. If the version numbers conform to the SemVer specification, we can use the NPM package SemVer for working on versions to help us compare version sizes and extract version information.

Npm also uses this tool to handle version-related work.

npm install semver
Copy the code
  • Compare the version number size
semver.gt("1.2.3".'9.8.7') // false
semver.lt("1.2.3".'9.8.7') // true
Copy the code
  • Determines whether the version number complies with the specification and returns the parsed version number that complies with the specification.
semver.valid("1.2.3") / / "1.2.3"
semver.valid('a.b.c') // null
Copy the code
  • Cast other version numbers to semver version numbers
semver.valid(semver.coerce('v2')) / / '2.0.0'
semver.valid(semver.coerce('42.6.7.9.3 - alpha')) / / '42.6.7'
Copy the code
  • Some other uses
semver.clean('= v1.2.3') / / "1.2.3"
semver.satisfies("1.2.3".'1. X | | > = 2.5.0 | | 5.0.0-7.2.3') // true
semver.minVersion('> = 1.0.0') / / '1.0.0'
Copy the code

These are the most common uses of semver. For more details, see the Semver documentation: github.com/npm/node-se…

2.4 Dependent version Management

We often see dependencies written differently in package.json:

"Dependencies" : {" signale ":" 1.4.0 ", "figlet" : "*", "react" : 16. "x", "table" : "~ 5.4.6", "yargs" : "^ 14.0.0"}Copy the code

The first three are easy to understand:

  • "Signale" : "1.4.0": Fixed version number
  • "figlet": "*": Any version (> = 0.0.0)
  • "react": "16.x": Matches major versions (> = 16.0.0 < 17.0.0)
  • "React" : "16.3 x": Matches major and minor versions (> = 16.3.0 < 16.4.0)

Let’s look at the next two versions that reference the ~ and ^ symbols:

  • ~: Installs to when a new version is obtained while installing a dependencyx.y.z 中 zThe latest version of the. That is, keep the latest version of the revision number without changing the major and minor versions.
  • ^: Installs to when a new version is obtained while installing a dependencyx.y.z 中 y 和 zAll are the latest versions. That is, keep the minor version number and revised version number as the latest version without changing the major version number.

The most common dependency in package.json files should be “yargs”: “^14.0.0”, because when we install the package using NPM Install Package, NPM installs the latest version by default, and then prefixes the installed version with the ^ sign.

Note that when the major version number is 0, it is considered an unstable version, which is different from the above:

  • Both major and minor versions are0: ^ 0.0. Z,~ 0.0. ZAre treated as fixed versions and do not change when installing dependencies.
  • The major version number is0: ^0.y.zPerformance and~0.y.zSame, only keep the revision number as the latest version.

The 1.0.0 version number is used to define the public API. Release 1.0.0 when your software is released to a formal environment or has a stable API. So, when you decide to release an official version of the NPM package externally, mark it as version 1.0.0.

2.5 Locking dependent Versions

The lock file

In real development, there are often strange problems with inconsistent dependencies, or in some scenarios where we don’t want dependencies to be updated, we recommend using package-lock.json for development.

Locking the dependency version means that the fixed version is installed each time we install the dependency without manually performing the update. Ensure that dependencies with consistent version numbers are used across the team.

Each time you install a fixed version, you do not need to calculate the dependent version range, which greatly speeds up the dependent installation time in most scenarios.

When using package-lock.json, make sure that the NPM version is above 5.6, because the package-lock.json processing logic was updated several times between 5.0 and 5.6, and has stabilized since 5.6.

The detailed structure of package-lock.json will be explained in a later section.

Periodically update dependencies

Our goal is to ensure that the dependencies used across the team are consistent or stable, rather than never updating them. In a real development scenario, we don’t need to install a new version every time, but we still need to upgrade the dependency version regularly to enjoy the problem fixes, performance improvements, and new feature updates brought by the upgrade of the dependency package.

Using NPM outdated helps us list dependencies that have not yet been upgraded to the latest version:

  • Yellow indicates that it does not fit within the semantic version range we specified – no upgrade required
  • Red indicates compliance with the specified semantic version range – requires an upgrade

NPM update updates all red dependencies.

2.6 Best practices for dependent version selection

release

  • When releasing a formal version of the NPM package to the outside world, mark its version as1.0.0.
  • After a package version is released, any changes must be released in the new version.
  • The version number is strictly followedMajor version. Minor version. Revision numberFormat name
  • Version number releases must be strictly incremental
  • Release major versions or major changes firstAlpha, beta, RCSuch as prior version

Dependency range selection

  • The main project relies on a number of sub-modules, all developed by team membersnpmPackage. In this case, you are advised to change the version prefix to~If the sub-dependency is locked, the dependency of the main project needs to be upgraded every time the sub-dependency is updated, which is very tedious. If the sub-dependency is fully trusted, it can be directly opened^Each time you upgrade to the latest version.
  • The main project runs indockerOnline, local dependencies are also being developed and upgraded indockerLock all dependent versions before release to ensure that there are no problems online after local child dependencies are released.

Keep dependencies consistent

  • Make sure thatnpmThe version of the5.6Above, make sure it is enabled by defaultpackage-lock.jsonFile.
  • Executed by the initializernpm inatallafterpackage-lock.jsonCommit to the remote repository. Don’t submit directlynode_modulesTo the remote warehouse.
  • Perform on a regular basisnpm updateUpgrade dependencies and commitlockThe file ensures that other members update their dependencies in sync and do not manually change themlockFile.

Depend on the change

  • Upgrade dependency: Modifiedpackage.jsonThe dependent version of the file is executednpm install
  • Degrade dependencies: Execute directlynpm install package@version(changepackage.jsonDependencies are not degraded)
  • Note that changes are committed after dependencieslockfile

NPM install

NPM install will probably go through several of the above processes, and this chapter will cover the implementation details of each process, its evolution, and why it is implemented this way.

3.1 Nested structure

As we all know, NPM install installs dependency packages into node_modules. Let’s see how NPM installs dependency packages into node_modules.

In earlier versions of NPM, NPM handled dependencies in a crude, recursive manner, installing dependencies into their respective node_modules in strict accordance with the package.json structure and the package.json structure of the child dependencies. Until a subdependency package no longer depends on another module.

For example, our module my-app now relies on two modules: buffer, ignore:

{
  "name": "my-app"."dependencies": {
    "buffer": "^ 5.4.3." "."ignore": "^ 5.1.4 ensuring",}}Copy the code

Ignore is a pure JS module and does not depend on any other modules, while Buffer relies on base64-JS and IeEE754 modules.

{
  "name": "buffer"."dependencies": {
    "base64-js": "^ 1.0.2"."ieee754": "^ 1.1.4." "}}Copy the code

So, after NPM install, the resulting module directory structure in node_modules looks like this:

The advantages of this approach are obvious: node_modules and package.json have a one-to-one hierarchical structure, and ensure that the directory structure is the same every time you install it.

However, imagine that if you rely on a lot of modules, your node_modules would be huge and nested very deeply:

  • In different levels of dependency, the same module may be referenced, leading to a great deal of redundancy.
  • inWindowsIn the system, the maximum length of file paths is 260 characters. Too deep nesting may cause unpredictable problems.

3.2 Flat Structure

To address these issues, NPM has made a major update in version 3.x. It changes the earlier nested structure to a flat one:

  • When installing a module, regardless of whether it is a direct dependency or a child dependency, install it in the first placenode_modulesThe root directory.

Still with the dependency structure above, we get the following directory structure after executing NPM install:

If we now rely on [email protected] in the module:

{
  "name": "my-app"."dependencies": {
    "buffer": "^ 5.4.3." "."ignore": "^ 5.1.4 ensuring"."base64-js": "1.0.1",}}Copy the code
  • When the same module is installed, check whether the installed module version conforms to the version range of the new module. If yes, skip it. If no, locate it in the current modulenode_modulesTo install the module.

At this point, we get the following directory structure after executing NPM install:

Accordingly, if we reference a module in the project code, the module lookup process is as follows:

  • Searches under the current module path
  • In the current modulenode_modulesSearch under path
  • In the superior modulenode_modulesDown path search
  • .
  • Until you find it in the global pathnode_modules

Suppose we again rely on a package buffer2@^5.4.3, which relies on the package [email protected], then the installation structure looks like this:

So NPM 3.x does not completely solve the problem of module redundancy in the previous version, and even introduces new problems.

Imagine that your APP assumes that it doesn’t rely on the [email protected] version, but you rely on both Buffer and Buffer2, which rely on different base64-JS versions. Since NPM install resolves dependencies in package.json in the order in which buffer and buffer2 are placed in package.json, node_modules dependencies are determined by the order in which they are placed:

Rely on buffer2 first:

Rely on buffer first:

In addition, in order to make it safe for developers to use the latest dependencies, we usually only lock the large version in package.json, which means that some dependencies may also change after the update of the small version, and the uncertainty of the dependency structure may bring unpredictable problems to the application.

3.3 the Lock file

In order to resolve the uncertainty of NPM install, package-lock.json was added in NPM 5.x, and the installation method is still flat NPM 3.x.

Package-lock. json locks dependencies, meaning that as long as you have package-lock.json in your directory, the node_modules directory must have exactly the same structure every time you execute NPM install.

For example, we have the following dependency structure:

{
  "name": "my-app"."dependencies": {
    "buffer": "^ 5.4.3." "."ignore": "^ 5.1.4 ensuring"."base64-js": "1.0.1",}}Copy the code

Package-lock. json is generated after NPM install:

{
  "name": "my-app"."version": "1.0.0"."dependencies": {
    "base64-js": {
      "version": "1.0.1"."resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.0.1.tgz"."integrity": "sha1-aSbRsZT7xze47tUTdW3i/Np+pAg="
    },
    "buffer": {
      "version": 5.4.3 ""."resolved": "https://registry.npmjs.org/buffer/-/buffer-5.4.3.tgz"."integrity": "sha512-zvj65TkFeIt3i6aj5bIvJDzjjQQGs4o/sNoezg1F1kYap9Nu2jcUdpwzRSJTHMMzG0H7bZkn4rNQpImhuxWX2A=="."requires": {
        "base64-js": "^ 1.0.2"."ieee754": "^ 1.1.4." "
      },
      "dependencies": {
        "base64-js": {
          "version": "1.3.1"."resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.3.1.tgz"."integrity": "sha512-mLQ4i2QO1ytvGWFWmcngKO//JXAQueZvwEKtjgQFM4jIK0kU+ytMfplL8j+n5mspOfjHwoAg+9yhb7BwAHm36g=="}}},"ieee754": {
      "version": 1.1.13 ""."resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.1.13.tgz"."integrity": "sha512-4vf7I2LYV/HaWerSo3XmlMkp5eZ83i+/CDluXi/IGTs/O1sejBNhTtnxzmRZfvOUqj7lZjqHkeTvpgSFDlWZTg=="
    },
    "ignore": {
      "version": "5.1.4 ensuring"."resolved": "https://registry.npmjs.org/ignore/-/ignore-5.1.4.tgz"."integrity": "sha512-MzbUSahkTW1u7JpKKjY7LCARd1fU5W2rLdxlM4kdkayuCwZImjkpluF9CM1aLewYJguPDqewLam18Y6AU69A8A=="}}}Copy the code

Let’s look at the above structure in detail:

The two outermost attributes name and version, like name and version in package.json, describe the current package name and version.

Dependencies are objects that correspond to the package structure in node_modules. The key of the object is the package name and the value is the package description:

  • version: Package version – This package is currently installed innode_modulesThe version in
  • resolved: Indicates the installation source of the package
  • integrity: packagehashValue, based onSubresource IntegrityTo verify whether the installed software package has been changed or invalid
  • requires: corresponds to the dependencies of child dependencies, and of child dependenciespackage.json 中 dependenciesHas the same dependencies.
  • dependenciesStructural and outerdependenciesThe structure is the same as the storage installed in the child dependenciesnode_modulesDependency packages in.

Note that not all child dependencies have the dependencies property. They only have this property if their dependencies conflict with those currently installed in node_modules in the root directory.

For example, recall the dependencies above:

The [email protected] version we rely on in my-app conflicts with base64-js@^1.0.2 in buffer, so [email protected] needs to be installed in node_modules of buffer package, Json corresponding to the Dependencies property of buffer in package-lock.json. This also corresponds to NPM’s flattening of dependencies.

So, according to the above analysis, package-lock.json files and node_modules directory structure are one to one, that is, the existence of package-lock.json in the project directory can keep the dependent directory structure generated by each installation the same.

Additionally, the use of package-lock.json in a project can significantly speed up dependency installation time.

NPM I –timing=true –loglevel=verbose NPM I –timing=true –loglevel=verbose Clean up the NPM cache before comparing.

Without using lock files:

Using lock files:

It can be seen that package-lock.json has already cached the specific version and download link of each package, so there is no need to go to the remote warehouse for query, and then directly enter the link of file integrity verification, which reduces a lot of network requests.

Use advice

When developing system applications, it is recommended that the package-lock.json file be submitted to the code repository to ensure that all team developers and CI sessions have the same dependencies that can be installed when NPM install is executed.

When developing an NPM package, your NPM package is dependent on other repositories. Due to the flat installation mechanism discussed above, if you lock the version of the dependency package, your dependency package cannot share the same semver scope with other dependencies, which can cause unnecessary redundancy. So package-lock.json files should not be published (NPM does not publish package-lock.json files either by default).

3.4 the cache

After you run the NPM install or NPM update command to download dependencies, the dependency package is installed in the node_modules directory and a copy of the dependency package is cached in the local cache directory.

You can run the NPM config get cache command to query the. NPM /_cacache directory in the home directory of a Linux user or Mac user by default.

In this directory, there are two more directories: Content-v2 and index-v5. The content-v2 directory is used to store the cache of the tar package, and the index-v5 directory is used to store the hash of the tar package.

During installation, NPM can generate a unique key corresponding to the cache record in the index-v5 directory according to the integrity, version, and name stored in package-lock.json to find the hash of the tar package. Then go to the cache tar package according to the hash to use directly.

We can find a package in the cache directory search test, index-v5 search package path:

grep "https://registry.npmjs.org/base64-js/-/base64-js-1.0.1.tgz" -r index-v5
Copy the code

Then we format json:

{
  "key": "pacote:version-manifest:https://registry.npmjs.org/base64-js/-/base64-js-1.0.1.tgz:sha1-aSbRsZT7xze47tUTdW3i/Np+pAg="."integrity": "sha512-C2EkHXwXvLsbrucJTRS3xFHv7Mf/y9klmKDxPTE8yevCoH5h8Ae69Y+/lP+ahpW91crnzgO78elOk2E6APJfIQ=="."time": 1575554308857."size": 1."metadata": {
    "id": "[email protected]"."manifest": {
      "name": "base64-js"."version": "1.0.1"."engines": {
        "node": "> = 0.4"
      },
      "dependencies": {},
      "optionalDependencies": {},
      "devDependencies": {
        "standard": "^ 5.2.2." "."tape": "4.x"
      },
      "bundleDependencies": false."peerDependencies": {},
      "deprecated": false."_resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.0.1.tgz"."_integrity": "sha1-aSbRsZT7xze47tUTdW3i/Np+pAg="."_shasum": "6926d1b194fbc737b8eed513756de2fcda7ea408"."_shrinkwrap": null."bin": null."_id": "[email protected]"
    },
    "type": "finalized-manifest"}}Copy the code

Above 6926 d1b194fbc737b8eed513756de2fcda7ea408 _shasum attributes for tar packet hash, namely the hash of the top 6926 first two layers for the cache directory, we go in this directory is indeed find the dependence on compressed package:

Before NPM V5, each cached module is stored directly as a module name in the ~/. NPM folder. The storage structure is {cache}/{name}/{version}.

NPM provides several commands to manage cached data:

  • npm cache add: The official explanation is that this order is mainlynpmUsed internally, but can also be used to manually add caching to a given package.
  • npm cache clean: Deletes all data in the cache directory. To ensure the integrity of the cache data, add--forceParameters.
  • npm cache verify: Verifies the validity and integrity of cached data and clears junk data.

Based on cached data, NPM provides the following offline installation modes:

  • --prefer-offline: Cache data is used preferentially. If no matching cache data is available, it is downloaded from the remote repository.
  • --prefer-online: Preferentially uses network data. If the network data request fails, it requests cache data. In this mode, the latest module can be obtained in time.
  • --offline: Does not request the network and directly uses the cache data. If the cache data does not exist, the installation fails.

3.5 File Integrity

We’ve talked a lot about file integrity, but what is file integrity verification?

Before downloading a dependency, we usually get the hash value calculated by NPM for that dependency. For example, if we execute the NPM info command, shasum(hash) follows the tarball:

After downloading the dependency package to the local PC, you need to ensure that no errors occur during the download. Therefore, you need to calculate the hash value of the file on the local PC after the download. If the two hash values are the same, ensure that the downloaded dependency is complete.

3.6 Overall Process

Ok, let’s summarize the above process as a whole again:

  • Check.npmrc files: priority is: project-level.npmrc files > user-level.npmrc files > global-level.npmrc files > built-in.npmrc files

  • Check for lock files in the project.

  • No Lock file:

    • fromnpmRemote repository retrieves package information
    • According to thepackage.jsonBuild the dependency tree, build the process:
      • When building a dependency tree, regardless of whether it is a direct dependency or a child dependency, it is placed first innode_modulesThe root directory.
      • When encountering the same module, judge whether the module version placed in the dependency tree conforms to the version range of the new module, if yes, skip, do not match, in the current modulenode_modulesPlace the module under.
      • Note that this step is only to determine the logical dependency tree, not the actual installation, which will be used to download or fetch the dependencies from the cache
    • Each package in the dependency tree is looked up in the cache in turn
      • No cache:
        • fromnpmRemote repository download package
        • Verify package integrity
        • The verification fails:
          • To download
        • Verification passed:
          • Copy the downloaded package tonpmThe cache directory
          • Unpack the downloaded package according to the dependency structure tonode_modules
      • Cache exists: Decompress the cache according to the dependency structure tonode_modules
    • Unzip the package tonode_modules
    • generatelockfile

  • Have lock file:

    • checkpackage.jsonWhether the dependent version inpackage-lock.jsonThere is a dependency conflict in.
    • If there is no conflict, you can skip the process of obtaining package information and building the dependency tree and start to search for package information in the cache. The following procedure is the same

The above procedure briefly describes the process of NPM install. This process also includes some other operations, such as executing some of the lifecycle functions that you define, You can run NPM install package –timing=true –loglevel=verbose to see the installation process and details of a package.

3.7 yarn

Yarn was released in 2016, when NPM was V3, and package-lock.json files were not available. As we mentioned above, unstable, slow installation, and other disadvantages were often teased by developers. At this point, YARN is born:

Here are the advantages of Yarn, which were very attractive at the time. Later, of course, NPM realized its problems and made many optimizations, including later optimizations (lock files, cache, default -s…). In, we can see the shadow of YARN more or less, so that yarn design is very good.

Yarn uses NPM V3 flat structure to manage dependencies. After dependencies are installed, a yarn.lock file is generated by default.

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


[email protected]:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/base64-js/-/base64-js-1.0.1.tgz#6926d1b194fbc737b8eed513756de2fcda7ea408"
  integrity sha1-aSbRsZT7xze47tUTdW3i/Np + pAg = base64 - js @ ^ 1.0.2: version"1.3.1"
  resolved "https://registry.yarnpkg.com/base64-js/-/base64-js-1.3.1.tgz#58ece8cb75dd07e71ed08c736abc5fac4dbf8df1"Integrity sha512 mLQ4i2QO1ytvGWFWmcngKO / / JXAQueZvwEKtjgQFM4jIK0kU + ytMfplL8j + n5mspOfjHwoAg + 9 yhb7bwahm36g = = buffer @ ^ 5.4.3:  version5.4.3 ""
  resolved "https://registry.yarnpkg.com/buffer/-/buffer-5.4.3.tgz#3fbc9c69eb713d323e3fc1a895eee0710c072115"
  integrity sha512-zvj65TkFeIt3i6aj5bIvJDzjjQQGs4o/sNoezg1F1kYap9Nu2jcUdpwzRSJTHMMzG0H7bZkn4rNQpImhuxWX2A==
  dependencies:
    base64-js "^ 1.0.2"
    ieee754 "^ 1.1.4." "Ieee754 @ ^ 1.1.4: version1.1.13 ""
  resolved "https://registry.yarnpkg.com/ieee754/-/ieee754-1.1.13.tgz#ec168558e95aa181fd87d37f55c32bbcb6708b84"Integrity sha512 + / - 4 vf7i2lyv/HaWerSo3XmlMkp5eZ83i CDluXi IGTs/O1sejBNhTtnxzmRZfvOUqj7lZjqHkeTvpgSFDlWZTg = = ignore @ ^ 5.1.4 ensuring:  version"5.1.4 ensuring"
  resolved "https://registry.yarnpkg.com/ignore/-/ignore-5.1.4.tgz#84b7b3dbe64552b6ef0eca99f6743dbec6d97adf"
  integrity sha512-MzbUSahkTW1u7JpKKjY7LCARd1fU5W2rLdxlM4kdkayuCwZImjkpluF9CM1aLewYJguPDqewLam18Y6AU69A8A==

Copy the code

Json file is similar to package-lock.json file, with some differences:

  • package-lock.jsonUsing thejsonFormat,yarn.lockA custom format is used
  • yarn.lockThe neutron-dependent version number is not fixed, meaning a separate oneyarn.lockCan’t find it outnode_modulesDirectory structure, also need andpackage.jsonFile to cooperate. whilepackage-lock.jsonAll you need is a file.

The caching policy of YARN looks similar to that before NPM V5. Each cached module is stored in an independent folder. The folder name contains the module name and version number. You can run the yarn cache dir command to view the directory where cached data is stored:

Yarn uses prefer-online mode by default. That is, network data is used preferentially. If the network data request fails, yarn requests cache data again.

reference

  • Juejin. Cn/post / 684490…
  • www.zhihu.com/question/30…
  • zhuanlan.zhihu.com/p/37285173
  • semver.org/lang/zh-CN/
  • Deadhorse. Me/nodejs / 2014…
  • Caibaojian.com/npm/files/p…

summary

I hope reading this article will help you:

  • To understandpacakge.jsonIn order to have a further view of the project engineering configuration
  • masternpmVersion management mechanism, can reasonably configure dependent versions
  • understandnpm installInstallation principle, can be used reasonablynpmCache,package-lock.json

If there are any mistakes in this article, please correct them in the comments section. If this article has helped you, please like it and follow it.

Want to read more quality articles, can follow my Github blog, your star✨, like and follow is my continuous creation power!

I recommend you to follow my wechat public account [Code Secret Garden] and push high-quality articles every day. We can communicate and grow together.