preface

In May, the application level Monorepo optimization scheme was shared, mainly explaining the existing problems and solutions of Monorepo (Yarn + Lerna). However, in this sharing, Pacakge publishing is not involved (mainly application app development during that period), and occasionally Pacakge development is also a relatively simple scenario (single package development/publishing), which can be done by USING NPM publish.

With subsequent development (mainly the migration of another warehouse within the team), the package development scenario was heavily weighted (warehouse lines in the millions and projects in excess of 100), but the multi-package release experience was not great, focusing on the following three areas:

  1. The release was quite different from Lerna, and the command documentation for Rush was rudimentary (so rudimentary, parameters tried many times) that it was hard to get started quickly;
  2. Release process is not standard enough, basically rely on the command line;
  3. Lack of standard development workflows.

The purpose of this sharing is to solve the above problems and find out better practices for Monorepo multi-package publishing scenarios in practice.

Workspace protocol (workspace:)

Before discussion to understand the Workspace protocol (Workspace), here in PNPM, for example, the following examples excerpted from the Workspace | PNPM

By default, PNPM links packages from the workspace if the package version available in the workspace matches the declared scope. For example, if [email protected] exists in monorepo and another project bar in Monorepo relies on “foo: ^1.0.0”, then the bar will use foo in the workspace. If the bar relies on “foo: 2.0.0”, PNPM will download [email protected] remotely for use by BAR, which introduces some uncertainty.

When using the Workspace protocol, PNPM will refuse to parse to anything other than the local workspace package. So if you set"Foo" : "the workspace: 2.0.0"The installation will fail because the workspace does not exist"[email protected]".

Package released

Basic operation

One of the advantages of Monorepo is the ease with which multiple packages can be distributed compared to the traditional single-repository, single-package publishing.

rush change

In Rush Monorepo, Rush Change is the starting point of the package distribution process, The output

–

.json is consumed by rush Version and Rush publish.

The changefile.json generation process is as follows:

  1. Detect the difference between the current branch and the target branch (usually master) and screen out items with changes (based ongit diffCommand);
  2. Ask for information (such as a version update policy and a brief description of what was updated) on the interactive command line for each filtered item;
  3. Based on the above informationcommon/changesChangefile.json corresponding to package is generated in the directory.

Note: The change type (type field) in the screenshot is None, not any of the major/ Minor /patch types. None means “roll these changes to the next patch, minor or major”, so in theory, If a project only has a change file of type “None”, it will neither consume the file nor upgrade the version.

The Type: None feature allows us to pre-merge packages that have been developed but do not need to follow the next release cycle into the master until the Pacakge has changefile.json that is not of Type None.

Rush Version and Rush publish

Rush Version or Rush publish –apply updates the version number based on the generated Changefile.json (i.e. Bump Version, following the Semver specification, The version number of the upper package of the released package may be updated, as described in the next section).

Rush publish — Publish will publish the corresponding package based on Changefile. json.

Rush’s distribution process is similar to Changesets, another popular Monorepo scenario distribution tool. Monorepo may be able to reuse this solution based on Changesets for pure PNPM 🥳.

  • 🦋 A way to manage your versioning and changelogs with a focus on monorepos
  • Changesets: popular Monorepo scenario package distribution tool

Cascade release

As mentioned earlier, when updating the version number, in addition to updating the version number of the package currently to be published, it may also update the version number of its upper-layer package, depending on how the upper-layer package refers to the current package in package.json.

As shown below, @modern-js/ plugin-tailwindCSS (the upper package) is introduced via “workspace:^1.0.0” in the form of @modern-js/utils (the lower package).

package.json(@modern-js/plugin-tailwindcss)

{
  "name": "@modern-js/plugin-tailwindcss"."version": "1.0.0"."dependencies": {
    "@modern-js/utils": "The workspace: ^ 1.0.0"}}Copy the code

package.json(@modern-js/utils)

{
  "name": "@modern-js/utils"."version": "1.0.0"
}
Copy the code

  • Rush will not update @modern-js/ plugin-tailwindCSS when updating the version number if @modern-js/utils is updated to 1.0.1. Since ^1.0.0 is compatible with 1.0.1, @modern-js/ plugin-tailwindCSS does not need to update the version number from a semantic point of view, To obtain @modern-js/[email protected], install @modern-js/[email protected] directly

  • Rush will update @modern-js/ plugin-tailwindCSS to 1.0.1 if @modern-js/utils is updated to 2.0.0. Because ^1.0.0 is not compatible with 2.0.0, update the @modern-js/ plugin-tailwindCSS version to 1.0.1 to reference the latest @modern-js/[email protected], The package.json content of @modern-js/plugin-tailwindcss is as follows:

{
  "name": "@modern-js/plugin-tailwindcss"."version": "1.0.1"."dependencies": {
   // The reference version number has also changed
    "@modern-js/utils": "The workspace: ^ 2.0.0." "}}Copy the code

The version number has been updated and needs to be published to NPM. Insert –include-all into rush publish and insert shouldPublish: If the version of true’s package is new than the NPM version, the package will be published.

This completes the semantically based cascading publishing.

An unexpected release

At the beginning of the Rush renovation project, projects in Monorepo always referred to each other with “workspace: *”, using the latest version in Monorepo.

This leads to two problems:

  1. When app goes online, it may go online with the package in the Development process (Trunk Based Development branch model, master is Trunk branch)
  2. Packages are shipped with unexpected releases because they are used"workspace: *", so the bottom package is updated, the top package must be released (to ensure that*The semantics of the)

Therefore, references between projects within MonorePO need to follow the following specification.

Reference standard

  1. Determine whether to use itworkspace:Reference the latest version in Monorepo
  2. If neededworkspace:, then please use"workspace: ^x.x.x"Instead of"workspace: *"Avoid meaningless releases (and of course consider actual dependencies, which should be used if packageA and packageB always need to be released together"workspace: *")

As the number of projects in MonorePO increases and workspace: references are used across all projects, a package update requires passive regression testing by all its internal access parties

There is no problem with CI to improve the master branch entry standards, but business scenarios are often too complex and require the involvement of testing students, unless the code quality of the team members and the testing quality are extremely high.

Or feature flag mechanism can be used for control, but manual control usually costs a lot and requires mature infrastructure schemes.

For projects that are not closely related to each other, they only exist in the same MonorePO at the physical level and do not need to pay attention to the latest version of each other and do not need to use workspace:.

For example, if the Babel/React/ModernJS suite is managed by the same monorepo, each suite will use the workspace internally: it is reasonable to enjoy the advantages of Monorepo project, and it is more appropriate to use the stable version of NPM directly for inter-project dependencies.

Of course, the business boundaries of open source projects are obvious, and when it comes to our business repository (one team per repository), there may be many different modules in it. In this case, using workspace: would be asking for trouble.

How do you decide whether to use workspace:?

For example, suppose I am the owner of the package bar and want to reference a package foo, using the following criteria:

If foo is updated, bar must be updated and tested and the iteration goes live at the same pace, then use workspace:, otherwise use the NPM remote version.

workflow

Note that:

  1. When function points are phased into the Trunk branch (master branch) in the development phase, the generated istype: noneChangefile.json, to avoid other packages being released with packages in development
  2. Because you need to generatetype: major/minor/patchChangefile. json is released in the test branch for the test package, so it will not be merged in the test phase. After acceptance, it will be merged for the official version release.

Assembly line detail

Test version

  1. Obtain the package to be published based on Changefile. json
  2. Install the dependencies of the target package as needed
    • rush install -t package1 -t package2
  3. Build the target package as needed
    • rush build -t package1 -t package2
  4. Rush Publish reads Changefile. json for version updates
    • rush publish –prerelease-name [canary.x] –apply
  5. Rush Publish publishes the package with the changed version number
    • rush publish –publish –tag canary –include-all –set-access-level public
  6. Synchronizes published information to relevant notification groups through robots

The official version

  1. Obtain the package to be published based on Changefile. json
  2. Install the dependencies of the target package as needed
    • rush install -t package1 -t package2
  3. Build the target package as needed
    • rush build -t package1 -t package2
  4. Pull a target branch to carry commits that are generated in the release process
  5. Rush Version consumes the changefile.json updated version number on the target branch pulled in the previous step and generates Changelog.md
    • rush version –bump –target-branch [source-branch] –ignore-git-hooks
  6. Rush update the lockfile on the target branch to avoid package.json inconsistency with the lockfile
  7. Rush publish publishes package to NPM
    • rush publish –apply –publish –include-all –target-branch [source-branch] –add-commit-details –set-access-level public
  8. Generates a Merge Request to Merge the target branch into the Master branch
    1. Deleting change files and updating change logs for package updates.
    2. Applying package updates.
    3. rush update.
  9. Synchronizing published information to relevant notification groups through robots (including Merge Request information, which needs to be merged in time)

Release speed

As you can see, the first three steps in the release process are consistent:

  1. Obtain the package to be published based on Changefile. json
  2. Install the dependencies of the target package as needed
    • rush install -t package1 -t package2
  3. Build the target package as needed
    • rush build -t package1 -t package2

However, when this solution first came to the ground, it used the simple and crude way of “fully installing Monorepo dependencies and fully building packages”.

Monorepo needs to address scale issues: projects get bigger, dependencies get slower to install, builds get slower, and test cases get slower to run.

“On demand” became the key word, PNPM was already very good as a package manager, and could even install dependencies on demand, but it was still lacking for large Monorepo, so we introduced Rush to solve engineering problems under MonorePO.

So the goal was clear: as Monorepo grew larger and larger, the complexity of the project remained stable. — Application level Monorepo optimization scheme

Prior to optimization, release closer to 12 minutes at a time, even if you release only one package with a console.log(” Hello world”) in it, and as projects grow, 12 minutes may be just the starting point. So “on demand” is back in the picture.

Rush will change the version number of the project to be released during the release process, so as long as the process is advanced and the project with the changed version number is obtained, the target parameters of the install and Build commands can be obtained.

The rush version code for @Microsoft /rush-lib is as follows:

function getVersionUpdatedPackages(params: { rushConfiguration: RushConfiguration; prereleaseName? :string;
}) {
  const { prereleaseName, rushConfiguration } = params;
  const changeManager: ChangeManager = new ChangeManager(rushConfiguration);

  if (prereleaseName) {
    const prereleaseToken = new PrereleaseToken(prereleaseName);
    changeManager.load(rushConfiguration.changesFolder, prereleaseToken);
  } else {
    changeManager.load(rushConfiguration.changesFolder);
  }
  // Change the package.json version number (in memory, the actual file is not changed)
  changeManager.apply(false);
  return rushConfiguration.projects.reduce((accu, project) = > {
    const packagePath: string = path.join(
      project.projectFolder,
      FileConstants.PackageJson,
    );
    // The version of the actual package.json
    const oldVersion = (JsonFile.load(packagePath) as IPackageJson).version;
    // Version number of package.json in memory
    const newVersion = project.packageJsonEditor.version;
    // Inconsistencies are our target items
    if(oldVersion ! == newVersion) { accu.push({name: project.packageName, oldVersion, newVersion });
    }
    return accu;
  }, [] as UpdatedPackage[]);
}
Copy the code

Auxiliary command

rush change-extra

The fault is caused by the access party’s lockfile. This command generates Changefile.json for the unchanged package so that it can be published.

By default, the rush change command compares the differences between the current branch and the master branch, identifies the items that have changed, and lets the developer generate the corresponding Changefile.json file through the interactive command line.

As mentioned in the previous “cascade release”, Rush can update the versions of related packages and release them according to the Semver specification. In the “workspace: ^ X.X.X “reference mode, the upper package will not be released unless the lower package is major updated.

The problem was this, the upper packages weren’t released, the lower packages were locked by the access side’s lockfile, and we needed a way to release packages that didn’t really need to be released (@jupiter/block-tools here), and that’s where Rush Change-Extra came in.

There is a greater need for a way to deeply update specified dependencies, but currently no solution has been found for the package manager dimension

conclusion

Based on the basic operation of Rush package delivery, this paper introduces some problems that will be encountered in the actual development process and gives the overall implementation scheme. Meanwhile, it optimizes the online release speed based on the idea of “on demand”.