Description of front-end package management mechanism

The blog link

from`package.json`File about

Package. json is a description file used in the front-end domain to describe information about a package. Only two fields are required:

name: Indicates the module namevalidate-npm-package-nameTo check whether the package name is valid because the package name is unique and NPM also provides itnpm view packageNameTo check whether the package name is occupied.
version: module version, NPM’s module version followsSemVerSpecification, i.e.,Major version Number. This version number. Revision number.AvailablesemverPackage to compare versions, extract version information and other operations.

Dependency management

Another important configuration in package.json is dependency management.

Dependency configuration fields are as follows:

dependencies: specifies the module on which the project will run
devDependencies: Some development-only packages can be managed with this field, dependencies that are not used at runtime, such as Eslint, etc. Keeping these dependencies to a number of times can avoid being installed in a production environment.
peerDependencies: Used to specify the version of the module on which the module you are developing depends, and the compatibility of the version of the dependencies installed by the user. After NPM3, the dependent version in peerDependencies is not forced to be installed. Instead, a WARN. Continue to install other dependencies
bundledDependencies: provides an array of package names that are packaged together at release time
optionalDependenciesIf your dependencies are optional and not available during installation, you can put them here and NPM will not report errors during installation.

Each dependency is controlled according to the rules of version control, which can be written as follows:

"XXX" : "1.0.0": Fixed version number
"xxx": "*": Any version number
"xxx": "16.x": indicates the major version number (here)< 17.0.0 16.0.0 < = version)
"XXX" : "16.3 x": matches major and minor versions (here)< 16.4.0 16.3.0 < = version)
"XXX" : "~ 1.0.0": Installs the latest version of a minor version without changing the major and minor versions
"XXX", "^ 1.0.0": Install the minor version and revised version as the latest version without changing the major version

This is the default version of NPM install.

In addition, when the major version number is 0, it is considered an unstable version, and versioning logic is different.

If both major and minor versions are 0, then~ 0.0. Zand^ 0.0. ZWill be treated directly as the fixed version.
If the major version is 0, then^0.y.zand~0.y.zSame, just keepzFor the latest version

Depend on the lock

In some scenarios, we don’t want dependencies to be updated, so we’ll generate a lock file in the same directory as package.json.

The purpose of this file is to install a fixed version each time without performing manual updates, thus ensuring consistent dependencies are used each time.

In the case of package-lock, ensure that the version of NPM is after 5.6 to achieve stable version control.

Upgrade depend on

Use NPM outdated to list the latest version of all dependencies in a table with package names rendered in different colors:

Yellow: does not fit our specified version range and does not need to be upgraded
Red: Meets the semantic version range we specified and needs to be updated

Then specifying NPM update automatically updates all red dependencies.

NPM package management mechanism

In the early days of NPM package management, NPM would simply install all dependencies into their respective node_modules in a recursive manner, strictly following the package.json structure and the package.json structure of the child dependencies. Until the subdependencies are no longer dependent on other modules.

While this management approach may seem very clear about the hierarchy of dependencies, there is also the problem of dependency redundancy. For example, if two child dependencies depend on the same project, the project will be installed twice and the whole project will have a very deep level of nesting.

To solve this problem, NPM smoothen the nesting structure in 3.x, meaning that when installing modules, direct and child dependencies as well as child dependencies are first installed under the root of node_modules.

If both A and B dependencies depend on project C and depend on different versions, then when installing this module:

The NPM checks whether the installed module meets the version range. If yes, the NPM skips
Nonconforming is in the current modulenode_modulesTo install the module.

At this point, we might get a directory structure like this:

Corresponding to this directory structure, when we search for a module, the corresponding search process is like this:

Searches under the path of the current module
In the current modulenode_modulesDown path search
In the superior modulenode_modulesDown path search
.
Until it’s in the global pathnode_modulesDown path search

In this mode, we solve part of the module dependency problem, but we also introduce new problems.

When NPM install is installed, modules and dependencies are parsed according to the order of modules in package.json, that is, the order of modules determines the resulting node_modules directory structure.

For example, if your A dependencies and B dependencies both depend on different versions of C, then node_modules, the c-dependent version of the first layer, depends on the order in which A and B are stored in package.json.

If B is followed by A:

If it is first A and then B:

In addition, if we lock only the larger version in package.json, the smaller versions of some dependencies will also change their dependency structure when they are updated.

In this case, we have to fix the node_modules hierarchy by specifying all dependencies in package-lock.json.

`package-lock.json`File structure

Note that not all child dependencies have a dependencies property. They only have this property if they conflict with a currently installed dependency.

In this way, the hierarchy in the lock file corresponds to the actual node_modules file to ensure that the same hierarchy is generated for node_modules each time NPM install is installed.

Another advantage of Lock is that the specific version and download link of each package are already cached, so you don’t need to go to the remote repository for query, and directly enter the file integrity check after installation.

NPM local cache

After installing dependencies using NPM Install, NPM caches these packages in the local cache directory.

The directory where the cache resides can be queried by using NPM config get cache.

NPM /_cacache directory, which contains two directories:

content-v2: storagetarThe package cache
index-v5: storagetarPackage the hash

During installation, NPM will generate a unique key corresponding to the cache record under index-v5 according to the intergrity, version, name in the lock file, so as to find the hash of the tar package. Then find the corresponding tar package from the hash cache and use it directly.

Prior to NPM V5, it was directly represented by structures like {cache}/{name}/{version}.

Common NPM commands for caching are:

npm cache clean: Clears the cache. To ensure the integrity of the cache data, it is usually added--forceparameter
npm cache verify: Verifies the validity and integrity of cached data and clears junk data

Based on caching, NPM provides an offline installation mode:

--perfer-offline: Preferentially use cache, if not download from remote repository
--perfer-online: Preferentially uses network data. If the network request fails, cache data is used
--offline: Does not request the network and directly uses the cache data. If the cache does not exist, the installation fails

Yarn Package Management Mechanism (YARn1)

Yarn was introduced in 2016, when NPM was in V3, a crude recursive mode. No flattening dependencies, no locking files. Yarn proposed its own solution to the defects of the original NPM and had a profound impact on subsequent updates to NPM.

Yarn has the following features:

generateyarn.lock: Clear dependency version and dependency structure, in any environment can get consistent dependency structure
Flat installation: Different versions of dependent packages are converted to a certain version range according to a certain policy
Using concurrent requests: Similar to concurrent connection pooling
Package caching was introduced.

What we are talking about here is the version management mode of YARN1.

`yarn.lock`Version of the lock

Unlike package-lock.json of NPM, yarn.lock uses a custom file format.

In addition, the version number of yarn.lock neutron dependence is not fixed, which means that yarn.lock alone cannot determine the directory structure of node_modules, and must be combined with package.json files. Package-lock. json, on the other hand, just needs to do it by itself.

The file contents of yarn.lock look like this:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY. # yarn lockfile v1 "@tootallnate/once@1": Version 1.1.2 resolved "Https://registry.yarnpkg.com/@tootallnate/once/-/once-1.1.2.tgz#ccb91445360179a04e7fe6aff78c00ffc1eeaf82" integrity Sha512 RbzJvlNzmRq5c3O09UipeuXno4tA1FE6ikOjxZK0tuxVv3412l64l5t1W5pj4 + rJq9vpkm/kwiR07aZXnsKPxw = = address @ > = 0.0.1, Address @ ^ 1.0.0: Version 1.1.2 resolved "Https://registry.yarnpkg.com/address/-/address-1.1.2.tgz#bf1116c9c758c51b7a933d296b72c221ed9428b6" integrity sha512-aT6camzM4xEA54YVJYSqxz1kv4IHnQZRtThJJHhUMRExaU5spC7jX5ugSwTaTgJliIgs4VhZOk7htClvQ/LmRA==Copy the code

As you can see in the yarn.lock file, the version of each package is a comma-concatenated array that represents the scope of all dependencies on that package. There are corresponding fields under each package, and these fields have meanings similar to those defined in package-lock.json:

Version: The exact version that matches the semantic version of the package
Resolved: Records the address of the packet. In addition, the value in the hash isshasum
Dependencies: Records the dependencies of the current package

Yarn raises all package dependencies to the top level. Dependencies with the same name that can be merged are separated by commas as one top-level key, and dependencies that cannot be merged are regarded as two independent top-level keys.

In the specific node_modules file, only the latest version of the package appears.

For example, our current project relies on two versions of the same package:

Fs - extra @ ^ 8.1.0: The version "8.1.0" resolved "Https://registry.yarnpkg.com/fs-extra/-/fs-extra-8.1.0.tgz#49d43c45a88cd9677668cb7be1b46efdb8d2e1c0" integrity sha512-yhlQgA6mnOJUKOsRUFsgJdQCvkKhcz8tlZG5HBQfReYZy46OwLcY+Zia0mtdHsOo9y/hP+CxMN0TU9QxoOtG4g== dependencies: Graceful-fs "^4.2.0" jsonfile "^4.0.0" universalify "^0.1.0" fs-extra@^9.0.1: The version "9.1.0" resolved "Https://registry.yarnpkg.com/fs-extra/-/fs-extra-9.1.0.tgz#5954460c764a8da2094ba3554bf839e6b9a7c86d" integrity sha512-hcg3ZmepS30/7BSFqRvoo3DOMQu7IjqxO5nCDt+zM9XWjb33Wg7ziNT+Qvqbuc3+gWpzO02JubVyk2G4Zvo1OQ== dependencies: At-lest-node "^1.0.0" graceful-fs "^4.2.0" jsonfile "^6.0.1" universalify "^2.0.0"Copy the code

These two versions cannot be combined, so YARN puts them both in the yarn.lock file.

But in the node_modules folder, the default is to place the [email protected] version of the module, and the [email protected] package is to be placed in the node_modules folder that references its dependencies.

The processing logic here is different from NPM.

Yarn Cache Management

Similar to NPM’s cache command, YARN provides several cache management commands:

yarn cache ls: Lists a list of packages currently cached
yarn cache dir: Displays the cached directory
yarn cache clean: Clear the cache

By default, the yarn cache directory is placed under the Library/Caches/ YARN/directory. You can view the details by calling the command.

Yarn cache management is more semantic:

NPM - yorkie d435214e12c51c2ae1093e54b6bb83d9 - integrity - - 2.0.0-92411912 NPM zeromq - b0b85e4152e98b8bb163f1db4a34280d44d9d0 - integrity - 5.2.8-94 NPM - zrender f4f8cc0f4742f02a6b8819550a6d13d64c5c - integrity - - 5.1.1-0515 NPM - zscroller eed68690808eedf81f9714014356b36cdd20f4 - integrity - - 0.4.8-69 NPM - zwitch - 1.0.5 - d11d7381ffed16b742f6af7b3f223d5cd9fe9920 - integrityCopy the code

Its cache directory is filled with folders like this, with file names starting with NPM – and consisting of package names, version numbers, and hash values.

Yarn defaults to perfer-online mode, which means that it tries to download from the remote repository and then tries to read from the cache if the connection fails. If you wish to use the offline first policy, the –offline parameter is also provided:

yarn add --offline
Copy the code

Yarn supports offline mirroring:

yarn config set yarn-offline-mirror ./npm-packages-offline-cache
Copy the code

Offline mirroring is different from caching. Caches are decompressed tar files downloaded from the registry. These caches are implemented by features-based tools. NPM has NPM caches, YARN has YARN caches, and the mechanism may be different under different versions of NPM. The tar package in an offline image is not affected by the tool and the tool version, and the stored file facilitates storage. More detailed steps can be found in the official documentation

(In fact, we see that under NPM’s caching mechanism, cached packages are stored in tar.)

PNPM package management mechanism

NPM version management after V3 and YARN1 version management have adopted the dependency leveling approach, which has the benefit of reducing the level of depth and dependency redundancy, but also introduces new problems:

The most serious problem is the issue of ghost dependencies, where modules can access packages they don’t depend on
The flattening dependent tree algorithm is very complex
Some packages must be copied to a projectnode_modulesIn the file

Let’s review the reasons for flattening packages:

Under the original NPM package management mechanism,node_modulesDeep nesting occurs.
There is a lot of redundancy of the same dependency in different dependency packages.

PNPM actually proposes another way around this problem, which is to centralize all packages under global management using hard links.

That is, the node_module folder retains the same file directory structure as the dependency graph, but avoids dependency redundancy and deep hierarchy.

In Linux, hard linking means linking through index nodes. In Linux file systems, files stored in disk partitions are assigned a number, called an inode number, regardless of their type. In Linux, multiple file names pointing to the same index node number is allowed. That is, Linux allows files to have multiple valid paths.

Based on this mechanism, PNPM prevents all dependencies from being stored tiled under the.pnpm file, so that each package can be found in a folder with this naming pattern:

.pnpm/<name>@<version>/node_modules/<name>
Copy the code

It’s officially called a virtual storage directory.

This storage structure avoids the long path problems caused by nested node_modules in previous versions of NPM-v3 and, unlike tiling in later versions of NPM-v3, preserves the true topology between dependent packages.

This approach to dependency has several benefits:

Maintains a strict node_modules hierarchy,node_modulesOnly real dependencies will exist in the folder
Because modules are shared globally, this solves the problem of project package size, and becausepnpmThe cache priority strategy makes the probability of packet reuse very high, and the speed of secondary installation on which the project depends is greatly improved.
No leveling dependencies introduce ghost dependencies

PeerDep dependency problem in PNPM

Suppose we now have two projects package-a and package-b:

package-aRely on[email protected]and[email protected]
package-bRely on[email protected]and[email protected]
lib-aThere is apeerDepsisLib - b @ ^ 1.0.0

If we were to install package-a and package-b at the same time in our project, we would need to smooth out all dependencies in.pnpm:

Package-a package-b [email protected] [email protected] [email protected]Copy the code

Package-a and package-b depend on lib-B versions respectively, but lib-b is also a dependency of lib-A, so which lib-B should be processed when importing lib-A? Obviously, you can’t just pick one, because both package-a and package-b depend on lib-A, and picking one or the other will result in an error.

So in this case, lib-a appears twice in PNPM. PNPM, once linked to [email protected] and once linked to [email protected].

That is, we get the following.pnpm directory:

Package - a package - b [email protected] + [email protected] [email protected] + [email protected] [email protected] [email protected]Copy the code

Yarn2, PnP mode

PnP mode is a bit more radical than PNPM’s global hardlink mode, which kills node_modules directly.

In PnP mode, Yarn maintains a static mapping table that contains:

What dependency packages are currently included in the dependency tree and the corresponding version groups
The dependencies of these dependency packages
The location of these dependency packages in the file system

This mapping table is stored as a.pnp.js file in the project root directory.

When a dependency is installed, YARN does not copy the dependency to the node_modules directory, but records the location of the dependency in the cache in.pnp.js. This avoids a lot of I/O and does not generate the node_modules directory in your project.

The advantages of PnP mode are:

Get rid of completelynode_modulesThe limits of
Improved module loading efficiency: pass.pnp.jsThe file can directly locate the address of the specific module, eliminating the process of searching the location of the module layer by layer
No longer constrained by the fact that different versions of modules with the same name cannot be in the same directory (a more complex hierarchy is created for this in NPM)

The biggest problem with PnP mode is compatibility. For example, many front-end tools rely on node_modules to find modules, for example:

The require of the Node
Module lookup for Webpack
TypeScript type declaration files
Plugin location mechanism for Babel, ESLint

, etc.

Deno package management

Deno is not a package management tool, but a new JS runtime framework designed to address the problems of node_modules.

The dependency import of Deno is to directly import the accessible path of resources, no matter url or local path, without any modular processing. Something like this:

import { serve } from "https://deno.land/[email protected]/http/server.ts";
const s = serve({ port: 8000 });
console.log("http://localhost:8000/");
for await (const req of s) {
  req.respond({ body: "Hello World\n" });
}
Copy the code

The path to each package contains the imported source + package name + version number + module, all tucked into a Url.

However, as the project volume increases, it is foreseeable that the introduction will become very complex and difficult to manage, and it is bound to introduce a set of dependency management mechanism.

The official recommendation is to create a dep.ts file locally and use the dep.ts file to import various local and remote project dependencies.

Like this:

/** * deps.ts * * This module re-exports the required methods from the dependant remote Ramda module. */
export {
  add,
  multiply,
} from "https://x.nest.land/[email protected]/source/index.js";
Copy the code

Then reference it in the project like this:

/** * example.ts */

import { add, multiply } from "./deps.ts";

function totalCost(outbound: number, inbound: number, tax: number) :number {
  return multiply(add(outbound, inbound), tax);
}

console.log(totalCost(19.31.1.2));
console.log(totalCost(45.27.1.15));

/** * Output ** 60 * 82.8 */
Copy the code

That is, remove the concept of package management tools and organize and manage module dependencies directly at the code level. This dependency management model, designed for decentralization, is a matter of opinion.

Refer to the link

NPM package Management (Full version)
Why do I now recommend PNPM over NPM/YARN?
Why should we use pnpm?
flat-node-modules-is-not-the-only-way
How peers are resolved
Managing dependencies | Deno Manual
Deno’s dependency management is a bit of a mess, but there is a cure

Description of front-end package management mechanism

frompackage.jsonFile about