Source: Sebastian Weber. Why Lerna and Yarn Workspaces is a Perfect Match for Building Mono-Repos — a Close Look at Features and Performance. March 18, 2019.
Original title: Why Lerna and Yarn Workspaces are the perfect match for building Monorepo — a close-up look at functionality and performance
Prologue by Translator: I’ve been looking for technical articles about Monorepo for a few days now, but I still don’t know what to read. However, I think this article is more straightforward to understand, suitable for me such a small white reference. The article introduces a longer section, focusing on the portal: the method of configuring MonorePO, and the time comparison of different methods
This post is my take on the topic of Mono-repo. After a brief introduction to Mono-Repos and its comparison to multi-Repos, I’ll move on to the tools for setting up Mono-Repos.
I’m not going to evaluate in detail which repository type is better in which case. However, the goal of this article is about Mono-Repos and how LERna, NPM, and Yarn Workspace can help you manage Mono-Repos. Using these tools in combination also makes sense. In particular, LERNA and Yarn Workspace can coexist peacefully in one project. How do you do this? We’ll find out soon enough.
What is Monorepo and how does it compare to multi-repO
Tools like Lerna and yarn workspace were a deciding factor, and as a result, managing your code base in a single REPo (aka mono-repo) has gained some traction in the last year or two. Many articles have been written and conferences given on the topic.
In short, mono-repo is a Git repository that houses multiple projects. Such projects are called workspaces or packages. Instead, use multiple warehouses with only one item per warehouse, called the multi-repo approach. Of course, a combination of the two approaches is possible. In my current job, we have multiple teams, each with its own repository. Some teams embrace the multi-repo approach, while others embrace the multi-repo motto. . In addition, there are teams that leverage both approaches, because the technology that is part of the repository is also a factor in the decision (for example, each Java microservice is part of its own Git repo).
To understand the difference between Mono-Repos and multi-Repos as well as the pros and cons, I recommend Markus Oberlehner’s article Monorepos in the Wild.
Monorepos tool chain
A Mono-repo hosts one or more projects or packages. These packages, which we call mini-Repos, can be versioned, built, and released independently. Therefore, each package contains its own package.json file, because each package is a complete project. Packages may have interdependent relationships. Managing these dependencies is done through symbolic links.
As we will see later, lerna and YARN workspaces give us the ability to build libraries and applications in a REPO without having to publish to NPM or other registries. The beauty behind these techniques is that they can find package dependencies by analyzing the package.json file located at the root of each project. As a result, these tools make it obsolete to manually create symbolic links or simply use “low-level” NPM links.
Lerna and YARN workspaces work together to improve the developer experience of managing multiple packages in Mono-repo.
Association between NPM, YARN, YARN workspace, and Lerna
I would like to make a messy note of how NPM, YARN, Yarn Workpaces, and Lerna relate to mono-Repos. Take a look at the Venn diagram below.
It depicts the three main characters and how they relate to each other. Oh, and don’t pay too much attention to the size of each circle. The purpose of this picture is just to give an impression of how these things are connected.
NPM (marked with 1) and YARN (2) are native package managers that share many features (3). For example, both use the package.json concept as a container for dependency management, which was introduced by NPM that year. More shared concepts and capabilities are dependency management, distribution, or the use of lock files to “freeze” dependent versions. There are even more features derived from NPM that YARN takes advantage of, such as publishing to the NPM registry.
One of the reasons yarn was created was because of performance issues — it took too long to install dependencies using NPM in large projects. Another aspect is the lack of features such as complex concepts for frozen versions, offline capabilities, or deterministic behavior in dependency resolution. Although, over time, many of the gaps in NPM have closed, and both technologies are now increasingly functional.
The things that still belong only to NPM (1) or YARN (2) are package-lock.json files or yarn.lock files, respectively. However, for us app developers, the different implementations of locking files don’t really matter. In fact, NPM and YARN are on a par when it comes to versioning.
One big feature that is unique to YARN is the YARN Workspace (4), which was added to YARN a year ago. It extends the yarn functionality with the native Mono-repo functionality. More on mono-repo’s capabilities in the next section.
Monorepo-related technologies – Which are native and which are user-space?
Consider the next diagram, which depicts how technologies in the Mono-REPo environment are interconnected.
Highlighted in red are the technologies that provide mono-repo functionality. All of these technologies are based on NPM or YARN. NPM and YARN do not provide advanced functions for building mono-repo except for NPM link or YARN Link.
Yarn Workspaces is the only representative that exposes mono-repo functionality natively. Lerna has been around for quite a long time, even before Yarn Workspaces existed.
Lerna uses semantic chaining for this purpose. ** It also allows the yarn workspace to be used, and then leaves the entire mono-repo aspect entirely to the functionality of the yarn workspace’s native implementation. ** In addition, Lerna provides sophisticated publishing and versioning capabilities, and can even publish projects independently. In short, Lerna provides a lot of functionality beyond mono-REPo management. On the other hand, the sole purpose of the YARN workspace is to simplify the Mono-repo workflow. So you don’t have to decide for either of them. Using LERNA with YARN workspaces makes complete sense.
Bolt is a relatively new project based on the YARN workspace. Inspired by Lerna, it aims to add more useful commands on top of this. However, I don’t have any experience because I haven’t finished getting Bolt running in my proving ground yet. Also, I realize that it has been submitted less recently. Therefore, I will not elaborate further in this article.
Different ways to configure Monorepo
I created a small repository to demonstrate the different variations. Simply clone the demo project REPO and switch branches for different variants. The readme.md file describes how to boot and use (that is, build and run virtual applications) specific variants. Another goal of this section and the demo project is to provide a simple testing ground to see how different variations work from different perspectives: what configuration steps are required, what steps are required to build and use subprojects (that is, packages), how dependency management works, or what the timing impact is on boot.
1. Do it yourself
I’ll skip this section, but feel free to check out the branch 1-do-it-yourself. Basically, you work with NPM links and have to create semantic links and manually install all subprojects. I hope you can imagine how cumbersome and impractical this would be for a real-world project.
2. learna + npm
To achieve automated support for manual tasks such as method 1, LERNA was introduced. You need to create a lerna.json file in the root directory. By convention, LERNA uses NPM by default.
As you can see in the next screenshot, you basically need to edit two files to get Lerna up and running: lerna.json and package.json. In lerna.json, you need to specify where Lerna is looking for packages.
To bootstrap all the subprojects, you need to perform lerna Bootstrap by calling the NPM script below.
$ npm run bootstrap
Copy the codeThe basic purpose of this command is to go to the root directory of all packages and perform the NPM installation. Take a look at these three bags and you’ll seeLerna told NPM to create a node_modules folder for each package.
3. lerna + yarn
This is the same as method 2, except that you must specify yarn as the client using the npmClient property in the lerna.json file. Boot is also performed by Lerna.
What’s the difference compared to method 1? It doesn’t really make any difference. It’s mostly a matter of taste, because the only difference is whether Lerna uses NPM or YARN as the dependency manager. The choice of method to answer this question boils down to the following questions.
- Which syntax do I prefer NPM run vs _YARN
- Should I stick to the quasi-standard or like Facebook’s efforts?
- Do I really care about lead time? If so, see the next chapter, which provides some performance benchmarks.
4. yarn workspaces
For this method, you don’t need LerNA. The YARN workspace has built-in mono-repo functionality. To use the YARN workspace, you need YARN 1.0 or higher. As you can see in the screenshot below, you don’t need a special configuration file. The package.json file in the root directory needs to be private and must have a “workspaces “property that tells YARN where to find the subproject (yarn calls it Workspaces).
To boot the project with all the workspaces, you just need to use YARN, because the YARN workspace itself provides this functionality.
$ yarn install
Copy the codeOr a little bit shorter
$ yarn
Copy the codeThis combines the two steps of methods 1 and 2: installing the root folder dependencies and bootstracking all packages.
One of the biggest differences compared to methods 1 and 2 is that the YARN workspace creates only a node_modules folder. All dependencies are promoted to the root directory. Note. This behavior can also be implemented in LERNA by using the -hoist flag (not practical YARN workspaces).
5. lerna + yarn workspaces
To configure LerNA with the YARN workspace, you must do the same configuration as described in method 4 in package.json in the root directory. However, you also need to provide a lerna.json file in the root directory, where you need to tell Lerna to use the YARN workspace. Unfortunately, you have to specify the location of the subproject unnecessarily in lerna.json. To boot the project, you do not need to use Lerna boot, you just need to use yarn Install as described in Method 4. There is little point in executing lerna Bootstrap at this point, as it simply calls YARN Install itself.
With this setup, LERNA completely hands off the dependent Settings and bootstrap workflow to the YARN workspace. So, it looks like we’ve configured more things to achieve the same effect as the previous method? So why do you use this method instead of method 4? Well, think about it — it makes perfect sense to use both lerna and YARN workspaces. They can coexist peacefully in the Mono-Repo project.
In this case:
- You only use the YARN workspace in the Mono-repo workflow.
- You use lerna’s utility commands to optimize the management of multiple packages, for example, by selectively executing NPM scripts for testing.
- You use Lerna to distribute packages because lerna versions and publish commands provide complex functionality.
Lerna is used to test and send packages. Yarn can do the rest of the work.
Lerna and yarn workspaces
The previous section gave you a quick look at how to set up Mono-Repos with different configurations. This section focuses more on the functionality of lerNA and YARN workspaces.
yarn workspaces
To date, yarn workspace is the only technology that provides native functionality for Mono-Repos. Unlike Lerna, where you don’t have to perform a separate step to boot package dependencies, YARN Install installs the root folder dependencies and then installs for each package.
Compared to LERNA, YARN Workspaces has no additional functionality other than dependency management for multi-project Settings. Because it is based on YARN, you have all the functionality of YARN at your fingertips.
To use the YARN workspace, Facebook has introduced some additional commands that only make sense in the case of Mono-Repos.
The following command will display the workspace dependency tree for your current project.
$ yarn workspaces info
Copy the codeThe next command allows you to run the selected YARN command in the selected workspace (that is, the package).
$ yarn workspace <package-name> <command>
Copy the codeFor example, react is added to a package/workspace named “awesome-package “as a development dependency using the following command (you can also use -d instead of -dev).
$ yarn workspace awesome-package add react --dev
Copy the codeHere is an example of removing dependencies from a particular package.
$ yarn workspace web-project remove some-package --save
Copy the codeIf you want to add a common dependency for all packages, go to the root directory of your project and use the -w (or -ignore-workshop-root-check) flag.
$ yarn add some-package -W
Copy the codeOtherwise, YARN will report an error.
With the following command, I added one of my packages (“awesome components”) to another package (“awesome app”) as a dependency. I found that you should specify a version number when adding a local package, otherwise YARN will try to find dependencies in the registry.
$YARN workspace @doppelmutzi/awesome-app add @doppelmutzi/[email protected] -dCopy the codeWith the workspace feature, YARN does not add dependencies to the node_modules directory of any package — only at the root level, where YARN promotes all dependencies to the root level. Therefore, YARN contains dependencies only once in a project.
You must take advantage of the noHoist feature of the YARN workspace to use incompatible third party dependencies that work in mono-REPo environments. You must specify this functionality in package.json in the project root directory, as shown in the following example.
// package.json
{
/ /...
"workspaces": {
"packages": ["packages/*"]."nohoist": [
"**/react-native"]}/ /...
}
Copy the codeSee the ConnectDotz demo for more information.
lerna
Like the YARN workspace, LERna adds mono-REP functionality to the front end project. However, as mentioned above, Lerna runs in “user space” and cannot be added to the native level.
If you configure LERna to use the YARN workspace, lerna will delegate the entire dependency management to the YARN workspace. If you configure Lerna to use NPM or YARN, lerna will use symbolic links to provide its own mono-repo functionality. In this case, you must use Lerna Bootstrap to initialize all package dependencies.
John Tucker wrote a great article on how to use Lerna’s commands to initialize projects and manage dependencies.
To install React as a dependency in all packages, you can use the following command.
$ lerna add react
Copy the codeIf you want to install React only as a dependency for a specific package, execute the following command.
$ lerna add react --scope my-package
Copy the codeIf you have React installed for every package, but only want to upgrade/downgrade to a specific version for a particular package, you can do this.
$lerna add [email protected] --scope my-packageCopy the codeLerna has several symbols. They constitute the options to be filtered in the lerna subcommand.
Consider the following NPM script named “test “. The following two shell commands show how to perform tests only on specific packages by using the -scope flag and globs. Lerna tries to perform YARN tests for each matched package.
// package.json { ... "Scripts ": {"test": "lerna exec YARN test"}... } $ yarn test --scope @my-company-services/* $ yarn test --scope @my-company/web-*Copy the codeAccording to the documentation, Lerna also provides the ability to promote shared dependencies to the root directory, as is the default behavior of the YARN workspace. So, you have to use the -hoist flag.
$ lerna add react -D --hoist
Copy the codeIf you use LERNA, one question is whether to choose NPM or YARN. As you can see from the “cheat sheet” in the previous section, you can easily switch between different package managers at will.
End workflow features and commands for advanced gameplay
Even if you choose the YARN workspace for dependency management, it is best to use LerNA. The reason is that Lerna provides useful commands to optimize the management of multiple packages. For example, with a lerna command, you can iterate over all or specific packages, running a series of operations (such as filtering, testing, and building) on each package. As such, it complements the YARN workspace and takes over the dependency management process.
Using Lerna for testing or linting within the root folder is faster than manually calling all operations from each package folder. John Tucker’s blog post details how to use LerNA for testing.
Version management and release are important development themes, and Lerna shines here as well. Lerna allows you to use two version management modes.
-
Fixed/locked mode. The version of each package can be managed at a single point (in the lerna.json file). If a package has been updated since its last release, it will be updated to the new version. Therefore, a major change in any one package will result in a new major version for all packages.
-
Standalone mode. The versions of software packages can be incremented independently of each other. Therefore, the “version” key in lerna.json needs to be set to “independent”. This approach provides more flexibility and is particularly useful for projects with loosely coupled components.
You can release packages that have changed since the last release.
$ lerna publish
Copy the codeIn standalone mode, there are different options to affect the version bump of the issued command. In addition to using the semver keyword, you can also use one of the following version upgrade flags: from-git or from-package.
The following command is published to the NPM registry using traditional submission criteria.
$ lerna publish --conventional-commits --yes
Copy the codeThe command above also generates a Changelog file. Depending on lerna’s documentation, there are different Changelog presets, such as Angular or Bitbucket. According to lerna’s documentation, there are different Changelog presets, such as Angular or Bitbucket. By the way, the yes flag skips all confirmation prompts.
In lerna.json, you can globally define that traditional submissions must be used instead of flags.
// lerna.json
/ /...
"command": {
"publish": {
"conventionalCommits": true."yes": true}}/ /...
Copy the code@jsilvax explains how traditional commits with LERNA work and how they are performed via CommitLint.
Because versioning and publishing is a complex topic, the section above shows only a small portion of the possibilities for Lerna. I won’t go into more detail because that would be beyond the scope of this article.
Time comparison of different methods
One of the main reasons people stick with YARN rather than NPM is the time it takes to install dependencies. Initially, YARN was developed because NPM took too long to install dependencies (in addition, NPM lacked some important functionality). At the same time, NPM made great efforts to close this gap in version 6.
Since you can implement Mono-repo in a number of ways, let’s take a look at how these different approaches behave. In the remainder of this section, I present the results of my performance experiments. I cloned the Babel project (circa October 2018) because it represents a real life Mono-repo with many packages (142 to be exact). Interestingly, the original setup of Babel leverages Lerna, which is configured to specify YARN as npmClient (with no YARN workspace) and to disable yarn lock file generation.
For each method (2-5), I did the following.
- I changed the configuration required for the corresponding methods (i.e., adjust package.json and lerna.json if necessary).
- I measured the time of installation dependencies and dedicated boot steps (if needed).
- I measured the time of three different use cases. For each use case, I made three measurements.
The above use case (UC) is.
1) I empty the NPM or YARN cache, I delete all node_modules folders, I delete all package-lock.json or yarn.lock files. 2) Cache exists, I deleted all the node_modules folder and deleted all package-lock.json or yarn.lock files. 3) Cache exists, package-lock.json or yarn.lock file exists, I delete all node_modules folder.
To clear the cache, I execute one of the following commands based on the NPM client I use.
$ npm cache clean --force
Copy the codeor
$ yarn cache clean
Copy the codeAs an aid to removing lock files and the node_modules folder, I added a script called cleanup.sh in the root of Babel.
find . -type f -name 'yarn.lock' -exec rm {} +
find . -type f -name 'package-lock.json' -exec rm {} +
find . -name "node_modules" -type d -prune -exec rm -rf '{}' +
Copy the codeBased on the use case, I ended up commenting out the first two lines.
To measure the execution time of the install and boot dependency steps, I used Gnomon. The following commands form examples of methods 2(using NPM’s LERna) and UC 1(empty cache, no node_modules folder, no lock file as a prerequisite) to show how I measure elapsed time.
$ npm cache clean --force && ./cleanup.sh && npm i | gnomon && npm run bootstrap | gnomon
Copy the codeBelow, you’ll find different measurements. I made these measurements over a period of time, so I played with different versions of Node, NPM, YARN, and LERna to see if different versions had different performance effects.
To switch between node and NPM versions, I used NVM. The following example first installs and uses V9 for Node and then installs V5.7.1 for NPM.
$NVM install v9 $NVM use v9 $NPM i-g [email protected]Copy the codeMethod 2 (lerna with NPM) — Node v10.12.0 / NPM v6.4.1 / LERNA 2.11.0
Method 2 (LERna with NPM) — Node v9.10.0 / NPM v5.6.0 / LERNA 2.11.0
Method 3 (LERna with YARN) – Node v10.12.0 / YARN 1.10.1 / LERna 2.11.0
Method 3 (LERna with YARN) – Node v9.10.0 / YARN 1.10.1 / LERna 2.11.0
Method 4 (YARN workspaces) – Node V10.12.0 / YARN 1.10.1
There is no need for the “bootstrap “step, because YARN Install does the work below.
Method 4 (YARN workspaces) – Node V11.2.0 / YARN 1.10.1
Method 5 (Lena with YARN workspaces) – Node V11.6.0 (NPM v6.5.0-Next-0)/YARN 1.12.3 / LERNA 3.8.0
In this way, I try to find out if using the YARN workspace as part of the LERNA configuration makes any difference to method 4. Since there is no need to boot with LerNA, the corresponding column is empty.
But as I expected, since LERNA does not participate in the dependency install/boot process, there is no difference from method 4.
Method 6 (Lerna + NPM CI + Audit) — Node v10.12.0 / NPM v6.4.1 / LERNA 3.4.3
In this approach, I use LERna and NPM CI, which constitute an alternative to NPM Install in a continuous integration environment. As of version 3 of Lerna, NPM CI is the default installation command. However, you can choose not to use it.
For this method, the package-lock.json file must exist. The node_modules folder should have been removed, otherwise you will receive a warning printed to the terminal. So UC 3 is out of the question.
Method 6 (LERna + NPM CI) — Node v9 / NPM v5.7.1 / LERNA 3.4.3
With this exact version of NPM, the NPM CI command is available, but there is no auditing capability. I want to test this setting to see if there is any performance impact without auditing dependencies. Again, UC 3 is out of the question in this case.
conclusion
Based on my measurements, I don’t see any significant difference in performance between the NPM and YARN workspace. Functionally, there is no difference. For me, which package manager to use is a matter of taste. In addition, they can be exchanged at any time or used in combination.
Currently, I prefer to use the YARN workspace as the Mono-REPo technique, because I like its ascending (reactive) functionality. On the other hand, using lerna and its -hoist flag is also ok. In my opinion, YARN Workpaces and Lerna are a good match. Configure LERNA to leave dependency management to the YARN workspace and use its utility commands.