[Recruitment] Bytedance Massive Star /TCM team continues to recruit front-end engineers, see the details below
Nowadays, many projects will use PNPM instead of NPM or YARN to do package management. As we all know, PNPM has a very attractive point in its propaganda that it can install our dependencies in a smaller and faster way. then
- Why is PNPM smaller and faster?
- Besides being smaller and faster, what other problems did he solve to get people interested?
To answer these questions, you need to understand some of the major “changes” in package management over the years, which brings us to today’s share.
Npm v1 & v2
At first, the design of NPM was very simple: there was a standard package manager for everyone to download and review, freeing developers from the days of “downloading resources from the Web, unpacking them manually and adding new projects”. So the first version of NPM used a very simple nested structure for version management.
Let’s assume that we have two projects:
- App1 depends on packageA and packageC, and both packageA and packageC depend on packageB.
- App2 depends on packageB and packageD, packageD depends on packageE, and packageE depends on packageF……
The nodule_packages directory structure for these two projects is as follows:
The problems with this are also obvious:
- The same dependencies are repeatedly installed in the project: for example, packageB is repeatedly installed in App1
- Dependency hell: like D, E, F…… in App2
- The same packages are repeatedly installed between different projects: for example, App1 and APP2 both have packageB installed
The Internet has some classic memes for this:
Npm v3
To solve this problem, NPM V3 completely rewrites the NPM installation program and flattens the main and child dependencies into node_modules, as in the above example.
The re-installation and dependency hell within the project seemed to have been solved nicely, but now some new problems arose:
1. Incomplete resolution of repeated dependencies
We assume that App1 has new dependencies on packageG and packageH, and packageG and packageH both depend on packageB v2.0, so node_modules may have the following dependencies:
A, C, G, and H depend on different versions of B. Therefore, the promotion of A subdependency only promotes the v1.0 version of packageB. The V2.0 version of packageB still hangs under G and H. Obviously, G and H still have two copies of the same Package EB V2.0 installed.
2. Uncertainty
Notice that I said “node_modules dependencies can have the following structure”, because node_modules can also have the following structure:
In this example, packageB V2.0 has been improved, with Bv1.0 under A and C.
So is it better to upgrade Bv1.0 or Bv2.0? Depending on the order in which the user installs, this uncertainty makes it extremely difficult to resolve the dependency issues of the project. The results of different installs may not be the same……
3. Implicit dependencies (aka ghost dependencies)
In the example above, although App1 does not directly declare that it references packageB, it can still be used normally in the project. The reason is that nPM3 tiled all dependencies into node_modules, so the require function could find it, which also introduced some risks to the project:
- Why is it possible to import a package that is not defined in package.json?
- Version uncertainties introduced: App1 requires A v1.0 or v2.0 version of packageB, which is dependent on the v1.0 or v2.0 version of packageB. An upgrade from V1.0 to V5.0 that App1 unknowingly relies on May cause unexpected problems for the project.
Yarn
In 2016, Facebook published this article on its website: Yarn: A New Package Manager for JavaScript. As the article begins, they encountered consistency, security, offline installation, and performance issues while using NPM.
Let’s see how YARN works:
- Lockfiles (yarn.lock) : records the versions of all dependencies that are installed. During installation, the version provided by the lockfile is preferred.
- Offline installation: When a dependency package is downloaded from the network, YARN stores it in the local global cache. The next download takes precedence over the global cache directory. If there is a dependency package, YARN copies it to the current directory.
- Performance issues: Parallel installation. Both NPM and YARN perform a series of tasks during installation. NPM executes each package in a queue: After the current package is installed, the next package is executed, while YARN synchronously executes all tasks.
The above uncertainty was resolved, along with faster installation, offline installation and other features. So when YARN was released, it received a lot of attention. On the same day, the official NPM blog published a Hello, YARN! . Congratulations on yarn’s open sourcing and praise the YARN team and Facebook for their contributions to the community and the NPM ecosystem. NPM V5 has also adopted YARN’s excellent locking and caching mechanisms.
At this point, the building of governance seems to have been built, but there are still two dark clouds hanging over it:
- Reuse between projects remains unresolved
- Implicit reliance on
Pnpm
1. Reuse between multiple projects
How do you solve reuse across multiple projects? Now that we have a local cache, can we not make a copy and rely on the project?
PNPM solves this problem by using hard links: Hard links, unlike soft links, point directly to the address of the original file on disk.
In the following example, App1 and App2 both rely on Bv1.0, and their node_modules take up 1MB of space, so it looks like they’re taking up 2MB of space together, but they’re actually pointing to the same disk space. So Bv1.0 takes up 1MB in both projects, not 2MB.
At the same time, this approach is also a good solution to the above mentioned “NPMV3 incomplete repeated dependency” problem.
Since most of the dependencies are hardlinked to node_modules in the project, it saves the cost of network downloads and copying disks, which is why PNPM is so fast:
2. Reuse of multiple versions of a dependency
For different versions of the same dependency package, only files that differ between versions will be stored. For example, if Bv1.0 contains 100 files and Bv2.0 releases only one file with changes (version2.js), PNPM local caching will only cache version2.js into storage, not save all files of the dependent package because of changes to one file.
3. Create a non-flat node_modules directory
How to resolve implicit dependencies? Since leveling would allow require to access dependencies correctly that it shouldn’t, why don’t I just cancel……
By default, PNPM’s node_modules level only has dependencies explicitly declared in package.json, and all other dependencies are placed under.pnpm.
At first glance, the.pnpm directory structure is similar to the node_modules directory of nPMv3, but it has a node_modules directory of nPMv1 &v2. In fact, these are all connected by soft links (symbolic links), as shown in the illustration on the website above. In this way, implicit dependencies can be eliminated perfectly, and the directory structure of current dependencies can be easily viewed without increasing storage space.
4. Monorepo support
Pnpm naturally supports monorepo projects. In addition to specifying workspace, Pnpm also provides a number of commands to facilitate the dependency management of projects under workspace, which will not be detailed here.
Existing problems
- The non-flat destructive structure and the need to use its own locking file, pnpm-lock.yaml, impose write costs on the migration;
- Soft connection compatibility, there are some scenarios that cannot be used;
- Dependencies of different applications are hard-linked to the same file, and if you modify a dependency file during debugging, you may inadvertently affect other projects.
Afterword.
In fact, NPM, YARN and PNPM have gradually iterated many functions and optimizations, and the speed is becoming more and more similar. Everyone is learning from each other and making progress from each other. It is this kind of atmosphere of pursuing perfection, honesty and clarity that makes the open source community better and better
[Wanted] Bytedance Mega Star /TCM team is looking for a front-end engineer
About us
Relying on the rich talent ecosystem and product capabilities of Douyin /TT, the huge Star Map front end team of Byte Commercialization effectively connects creators and advertisers, inspires the marketing value of high-quality creation, and grows from a small project to an independent business of the group in three years. We look forward to your joining us!
How to help creators realize better? _bilibili bilibili
For more details about the team/business, please refer to the front-end paradise or overtime hell? Tell me about my year in Byte.
If you are
-
Advocate freedom, and want to do excellent things with excellent people;
-
Love front-end, love technology, the pursuit of perfection;
-
Self-driven, responsible;
-
King of Glory players, the team everyone king section, come and join us on the score!
Welcome to join us, we have a large number of recruitment of front-end engineers, club/school recruitment/intern, ability level is not limited, work in Shanghai/Beijing/Hangzhou/Mountain View!
Look forward to the excellent you to join the big star map team!
Consulting with
If you want to know more about the business, the team and the position, you can add Cun Zhi wechat: Island205, email: [email protected], and TL direct communication!
Resume direct investment
Remember to note the directional big star map front end team oh!
Front-end Development Engineer (Shanghai) — Creativity and Ecology
job.toutiao.com/s/NSMLh2N
Star Map Platform front-end development engineer (Hangzhou) – Creativity and Ecology
job.toutiao.com/s/NSMYNmm
Frontend Engineer, TikTok Ads Creative & Ecosystem
job.toutiao.com/s/NS6KNHt
Frontend Engineer, TikTok Creator Marketplace (Los Angeles)
job.toutiao.com/s/NSMjkrH
Front-end Development Intern (Shanghai/permanent) – Creativity and Ecology
job.toutiao.com/s/NS65GKU
Front-end Development Intern (Hangzhou/permanent) – Creativity and Ecology
job.toutiao.com/s/NS6K9n2