This is the fourth day of my participation in the November Gwen Challenge. See details: The last Gwen Challenge 2021.

Writing in the front

Webpack can be regarded as a mainstay in front-end building tools, such as daily business development, front-end infrastructure tools, advanced front-end interviews… It shows up in every scene.

You may be confused about its internal implementation mechanism, and still do not understand the meaning and application of each parameter based on the Webpack Plugin/Loader API in your daily work.

All of this is essentially due to the lack of a clear understanding of the Webpack workflow, which leads to the so-called “don’t know what to do with the API” development.

In this article, we will start from the perspective of how to implement module analysis project packaging, using the most popular, the most concise, the most clear code to take you to uncover the mystery behind Webpack, take you to implement a simple version of Webpack, from now on for any WebPack-related low-level development know the chest.

We’ll just cover the “dry stuff” here, taking you through the webPack workflow with the most accessible code.

I want you to know the basics

  • Tapable

The Tapable package is essentially a library for us to create custom events and trigger custom events, similar to the EventEmitter Api in Nodejs.

The plug-in mechanism in Webpack is based on Tapable implementation decoupled from the packaging process, and all forms of plug-ins are based on Tapable implementation.

  • Webpack Node Api

For learning purposes, we will focus on the Webpack Node Api process. In fact, the NPM run build command we use daily in the front end is also used to call the bin script to call the Node Api to execute compilation and packaging through environment variables.

  • Babel

AST analysis within Webpack also relies on Babel for processing, if you are not familiar with Babel. I encourage you to take a look at these two articles “Front-end Infrastructure” that take you through the world of Babel and # From Tree Shaking into the world of Babel plugin developers.

I’ll go into more detail about how these things work in Webpack later, but I’d love to see you click on the documentation above to get a sense of what you can do before reading this article.

The process to comb

Before we begin, let’s take a look at the whole packaging process.

This is just an overview of the whole process. Now you don’t need to think about what happens in each step in detail. We will take you through them step by step in the following steps.

As a whole, we will analyze the Webpack packaging process from the above five aspects:

  1. Initialization parameter phase.

    This step will read the corresponding configuration parameters from our configured webpack.config.js and merge the parameters passed in the shell command to get the final package configuration parameters.

  2. Start the compilation preparation phase

    In this step we return a Compiler method by calling the webpack() method, create our Compiler object, and register each webpack Plugin. Find the entry code in the configuration entry and call the compiler.run() method to compile.

  3. Module compilation stage

    The file is analyzed from the entry module and loaders of the matching file are called to process the file. At the same time, the module is analyzed and compiled recursively.

  4. Complete compilation phase

    After the recursion is complete, each reference module is processed by loaders and the interdependence between modules is obtained.

  5. Output file stage

    Unpack module dependencies and output the processed files to ouput’s disk directory.

Let’s explore in detail what happens at each step.

Create a directory

To do a good job, he must sharpen his tools. Let’s start by creating a good directory to manage the Packing tools we need to implement.

Let’s create a directory like this:

  • webpack/coreStore what we’ll implement ourselveswebpackCore code.
  • webpack/exampleStore the instance items we will use for packaging.
    • webpack/example/webpak.config.jsConfiguration file.
    • webpack/example/src/entry1The first entry file
    • webpack/example/src/entry1Second entry file
    • webpack/example/src/index.jsModule file
  • webpack/loadersStore our customizationsloader.
  • webpack/pluginsStore our customizationsplugin.

Initialization parameter phase

There are usually two ways to pass packaging parameters to WebPack in the daily use phase. Let’s look at how to pass parameters first:

CliThe command line passes arguments

In general, when we call the webpack command, we sometimes pass in certain command-line arguments, such as:

webpack --mode=production
#Call the webpack command to perform the packaging while passing mode as Production
Copy the code

webpack.config.jsPassing parameters

The other way, I believe, is more cliche.

We use webpack.config.js in the project root directory to export an object for the webpack configuration:

const path = require('path')

// Import loader and plugin...
module.exports = {
  mode: 'development'.entry: {
    main: path.resolve(__dirname, './src/entry1.js'),
    second: path.resolve(__dirname, './src/entry2.js'),},devtool: false.// Base directory, absolute path, used to parse entry points and loaders from the configuration.
  In other words, all relative paths of entry and Loader are relative to this path
  context: process.cwd(),
  output: {
    path: path.resolve(__dirname, './build'),
    filename: '[name].js',},plugins: [new PluginA(), new PluginB()],
  resolve: {
    extensions: ['.js'.'.ts'],},module: {
    rules: [{test: /\.js/,
        use: [
          // There are three ways to use your own loader
          path.resolve(__dirname, '.. /loaders/loader-1.js'),
          path.resolve(__dirname, '.. /loaders/loader-2.js'),],},],},};Copy the code

At the same time the configuration file is we need as instance instance configuration under the project example, let’s modify the example/webpack config. The contents of the js for the configuration.

Of course, you don’t need to understand the loader and plugin here, but we will implement these things gradually and add them to our packaging process.

Implement the merge parameter phase

This step, let’s really start implementing our WebPack!

First let’s create a new index.js file under webpack/core as the core entry file.

Create a new webpack.js file under webpack/core to implement the webpack() method.

First, it is clear that the Compiler object is obtained using the Webpack () method in the NodeJs Api.

At this point, let’s supplement the logic in index.js with the original Webpack interface format:

  • We need awebpackMethod to execute the call command.
  • At the same time we introducewebpack.config.jsConfiguration file passingwebpackMethods.
// index.js
const webpack = require('./webpack');
const config = require('.. /example/webpack.config');
// Step 1: Initialize parameters Synthesize parameters from the configuration file and shell parameters
const compiler = webpack(config);
Copy the code

Well, it looks good. Let’s implement webpack.js:

function webpack(options) {
  // Merge parameters to get merged parameters mergeOptions
  const mergeOptions = _mergeOptions(options);
}

// Merge parameters
function _mergeOptions(options) {
  const shellOptions = process.argv.slice(2).reduce((option, argv) = > {
    // argv -> --mode=production
    const [key, value] = argv.split('=');
    if (key && value) {
      const parseKey = key.slice(2);
      option[parseKey] = value;
    }
    return option;
  }, {});
  return{... options, ... shellOptions }; }module.export = webpack;
Copy the code

One extra note we need to make here

In the webpack file, you need to export a method called Webpack and accept configuration objects passed in from outside. This is what we talked about above.

Of course, our logic for merging arguments is to eventually merge the object passed in from the outside with the arguments passed in when the shell is executed.

In Node Js we can use process.argv.slice(2) to retrieve arguments passed in shell commands, such as:

Of course, the _mergeOptions method is a simple method to merge configuration parameters, I believe for everyone is a piece of cake.

Congratulations 🎉, a journey of a thousand miles begins with a single step. At this stage we have completed the first step in the packaging process: merging configuration parameters.

Compilation phase

After we have the final configuration parameters, we need to do the following things in the webpack() function:

  • The Compiler object is created with the parameters. We see a Compiler object returned by calling the Webpack (options) method in the official case. And package the code launched by the compiler.run() method at the same time.

  • Register the WebPack Plugin we defined.

  • Find the corresponding package entry file based on the incoming configuration object.

createcompilerobject

Let’s complete the logical code in index.js first:

// index.js
const webpack = require('./webpack');
const config = require('.. /example/webpack.config');
// Step 1: Initialize parameters Synthesize parameters from the configuration file and shell parameters
// Step 2: Call Webpack(options) to initialize the Compiler object
The webpack() method returns a Compiler object

const compiler = webpack(config);

// Call the run method for packaging
compiler.run((err, stats) = > {
  if (err) {
    console.log(err, 'err');
  }
  // ...
});
Copy the code

As you can see, the core compilation implementation lies in the Compiler.run () method returned by the webpack() method.

Step by step, let’s refine the webpack() method:

// webpack.js
function webpack(options) {
  // Merge parameters to get merged parameters mergeOptions
  const mergeOptions = _mergeOptions(options);
  // Create the Compiler object
  const compiler = new Compiler(mergeOptions)
  
  return compiler
}

// ...
Copy the code

Let’s also create a new compiler. Js file in the webpack/core directory as the compiler core implementation file:

// compiler.js
// Compiler class for core Compiler implementation
class Compiler {
  constructor(options) {
    this.options = options;
  }

  // The run method starts compilation
  // Meanwhile, the run method accepts an externally passed callback
  run(callback){}}module.exports = Compiler
Copy the code

At this point, our Compiler class first builds a basic skeleton code.

So far, we have:

  • Webpack /core/index.js as the entry file for the packaging command, this file references our own implementation of Webpack and external webpack.config.js(options). Call webpack(options).run() to start compiling.

  • The webpack/core/webpack.js file currently handles the merging of parameters and passing in the merged parameter New Compiler(mergeOptions), as well as returning the created Compiler strength object.

  • Webpack /core/ Compiler. At this point, our Compiler is just a basic skeleton with a run() startup method.

writePlugin

Remember that we used two plugins in webpack.config.js: pluginA and pluginB. Let’s implement them in turn:

Before we can implement Plugin, we need to improve the Compiler method:

const { SyncHook } = require('tapable');

class Compiler {
  constructor(options) {
    this.options = options;
    // Create plugin hooks
    this.hooks = {
      // The hook to start compiling
      run: new SyncHook(),
      // Execute asset to output directory (before writing file)
      emit: new SyncHook(),
      // Perform all compilation execution at compilation completion
      done: new SyncHook(),
    };
  }

  // The run method starts compilation
  // Meanwhile, the run method accepts an externally passed callback
  run(callback){}}module.exports = Compiler;
Copy the code

Here, we create a property hooks in the Compiler class’s constructor, which has values of run, emit, and done.

The values of these three attributes are tapable’s SyncHook method, which we mentioned above. Essentially, you can simply think of SyncHook() as an Emitter Event class.

When we return an instance of an object with new SyncHook(), we can add event listeners to the object using the this.hook.run.tap(‘name’,callback) method. All tap-registered events are then executed through this.hook.run.call().

Of course, webpack real source code, there are a lot of hooks. And there are synchronous and asynchronous hooks. We are trying to clarify the process here, so we just listed three common and simple synchronous hooks.

At this point, we need to understand that we can register the hooks with compiler.links.run.tap on the instance object returned by the Compiler class.

Let’s cut back to webpack.js and fill in the logic for plug-in registration:

const Compiler = require('./compiler');

function webpack(options) {
  // Merge parameters
  const mergeOptions = _mergeOptions(options);
  // Create the Compiler object
  const compiler = new Compiler(mergeOptions);
  // Load the plug-in
  _loadPlugin(options.plugins, compiler);
  return compiler;
}

// Merge parameters
function _mergeOptions(options) {
  const shellOptions = process.argv.slice(2).reduce((option, argv) = > {
    // argv -> --mode=production
    const [key, value] = argv.split('=');
    if (key && value) {
      const parseKey = key.slice(2);
      option[parseKey] = value;
    }
    return option;
  }, {});
  return{... options, ... shellOptions }; }// Load the plug-in function
function _loadPlugin(plugins, compiler) {
  if (plugins && Array.isArray(plugins)) {
    plugins.forEach((plugin) = >{ plugin.apply(compiler); }); }}module.exports = webpack;
Copy the code

Here, after creating the Compiler object, we call the _loadPlugin method to register the plug-in.

Those of you who have been involved in webpack plug-in development, more or less, probably have. Every WebPack plug-in is a class (of course classes are essentially syntactic sugar of funciton), and every plug-in must have an Apply method.

The Apply method accepts a Compiler object. All we did above was call the apply method of the plugin passed in and pass in our Compiler object.

Here I ask you to remember the above process, daily we writewebpack pluginTime is essentially operationcompilerObject thus affecting the packaging results to proceed.

Maybe you don’t quite understand the meaning of this sentence at this time, I will reveal the answer after we finish the whole process.

Let’s write these plugins:

If you don’t know about plug-in development, you can take a look at the official introduction. It’s not very difficult. I strongly suggest that if you don’t know about plug-in development, you can take a look and come back.

First let’s create the file:

// plugin-a.js
/ / A plug-in
class PluginA {
  apply(compiler) {
    // Register synchronization hooks
    // The compiler object is the instance created by new Compiler()
    compiler.hooks.run.tap('Plugin A'.() = > {
      / / call
      console.log('PluginA'); }); }}module.exports = PluginA;
Copy the code
// plugin-b.js
class PluginB {
  apply(compiler) {
    compiler.hooks.done.tap('Plugin B'.() = > {
      console.log('PluginB'); }); }}module.exports = PluginB;
Copy the code

Tap creates a SyncHook instance with tapable and then registers the event with the TAP method.

That’s right! Indeed, the webPack plug-in is essentially a publish-subscribe model, listening for events in compiler. It then packages the events that trigger listening during compilation to add some logic to affect the packaging results.

We subscribe to the corresponding events via tap on the Apply method of each plug-in in the compile preparation phase (that is, when we call webpack()), and publish the corresponding events when we reach a certain stage of compilation to tell the subscriber to execute the listening events. Thus, the corresponding plugin can be triggered in different life cycles of the compile phase.

Therefore, you should be aware that when we were developing the WebPack plug-in, the Compiler object stored all the related properties of the package, such as the configuration of the Options package, as well as various properties we will talk about later.

Looking forentryThe entrance

After that, most of our work will be in Compiler.js to implement the compiler class that implements the packaged core process.

Any time you pack, you need an entry file, so let’s move on to the actual package compilation phase. The first thing we need to do is find the corresponding entry file based on the entry profile path.

// compiler.js
const { SyncHook } = require('tapable');
const { toUnixPath } = require('./utils');

class Compiler {
  constructor(options) {
    this.options = options;
    // Relative path and path Context parameters
    this.rootPath = this.options.context || toUnixPath(process.cwd());
    // Create plugin hooks
    this.hooks = {
      // The hook to start compiling
      run: new SyncHook(),
      // Execute asset to output directory (before writing file)
      emit: new SyncHook(),
      // Perform all compilation execution at compilation completion
      done: new SyncHook(),
    };
  }

  // The run method starts compilation
  // Meanwhile, the run method accepts an externally passed callback
  run(callback) {
    // The plugin that starts compiling is triggered when the run mode is called
    this.hooks.run.call();
    // Get the entry configuration object
    const entry = this.getEntry();
  }

  // Get the path of the entry file
  getEntry() {
    let entry = Object.create(null);
    const { entry: optionsEntry } = this.options;
    if (typeof entry === 'string') {
      entry['main'] = optionsEntry;
    } else {
      entry = optionsEntry;
    }
    // Change entry to an absolute path
    Object.keys(entry).forEach((key) = > {
      const value = entry[key];
      if(! path.isAbsolute(value)) {// Converts to an absolute path with a unified path separator of /
        entry[key] = toUnixPath(path.join(this.rootPath, value)); }});returnentry; }}module.exports = Compiler;
Copy the code
// utils/index.js
/** ** The unified path separator is used to facilitate the generation of module ids *@param {*} path
 * @returns* /
function toUnixPath(path) {
  return path.replace(/\\/g.'/');
}
Copy the code

In this step we get the absolute path to the entry file through the options.entry process.

Here are a few tips to note:

  • this.hooks.run.call()

After subscribing to the Compiler instance object for each incoming plug-in in our _loadePlugins function, we actually start compiling when we call the run method. This is the phase where we need to tell the subscriber to publish the subscription to start executing. At this point we execute all tap listening methods on run via this.links.run.call () to trigger the corresponding plugin logic.

  • this.rootPath:

In the external webpack.config.js we configured a context: process.cwd(), which is also process.cwd() by default in real Webpack.

And a detailed explanation of that you can see Context here.

In short, this path is the directory path to which our project is launched, and any relative path in entry and loader is the relative path to the context parameter.

Here we use this.rootPath in the constructor to hold the variable.

  • toUnixPathTools and methods:

The paths for separating files are different in different operating systems. Here we use \ to replace the // in the path for the module path. In the future, we will use the module’s path relative to rootPath as the unique ID of each file, so path separators will be treated uniformly here.

  • entryProcessing methods:

There are many different entry configurations available in Webpack. Here we consider two common configurations:

entry:'entry1.js'

// Essentially this code in Webpack is converted to
entry: {
    main:'entry1.js
}
Copy the code
entry: {
   'entry1':'./entry1.js'.'entry2':'/user/wepback/example/src/entry2.js'
}
Copy the code

Either way, the getEntry method is converted to {[module name]:[module absolute path]… }, the geEntry() method is actually quite simple, so I won’t bother implementing it too much here.

At this point, we get an object with entryName as key and value as entryAbsolutePath using getEntry. Now let’s start compiling from the entry file.

Module compilation stage

Above we talked about the preparation for the compile phase:

  • Directory/file base logic supplement.
  • throughhooks.tapregisteredwebpackThe plug-in.
  • getEntryMethod to get objects for each entry.

Let’s continue to refine compiler.js.

During module compilation, we need to do the following events:

  • Analyze the entry file according to the entry file path, and match the corresponding entry fileloaderProcess the entry file.
  • willloaderProcess the finished entry file for usewebpackCompile.
  • Analyze the import file dependencies and repeat the previous two steps to compile the corresponding dependencies.
  • If the nested file has a dependency file, the dependency module is recursively called for compilation.
  • After the recursive compilation is complete, assemble the modules one by onechunk

First, we’ll add some logic to the compiler.js constructor:

class Compiler {
  constructor(options) {
    this.options = options;
    // Create plugin hooks
    this.hooks = {
      // The hook to start compiling
      run: new SyncHook(),
      // Execute asset to output directory (before writing file)
      emit: new SyncHook(),
      // Perform all compilation execution at compilation completion
      done: new SyncHook(),
    };
    // Save all entry module objects
    this.entries = new Set(a);// Save all dependent module objects
    this.modules = new Set(a);// All code block objects
    this.chunks = new Set(a);// Store the output file object
    this.assets = new Set(a);// The name of the output of this compilation
    this.files = new Set(a); }// ...
 }
Copy the code

Here we store the corresponding resource/module objects generated at compile time by adding column properties to the Compiler constructor.

About Entries \modules\chunks\assets\ Files These Set objects are properties that run through our core packaging process, each of which is used to store different resources at the compile stage and ultimately generate compiled files through the corresponding properties.

Analyze the import file based on the import file path

We can already pass this.getentry () in the run method; Get the corresponding entry object ~

Next, let’s start from the entry file to analyze the entry file!

class Compiler {
    // The run method starts compilation
  // Meanwhile, the run method accepts an externally passed callback
  run(callback) {
    // The plugin that starts compiling is triggered when the run mode is called
    this.hooks.run.call();
    // Get the entry configuration object
    const entry = this.getEntry();
    // Compile the entry file
    this.buildEntryModule(entry);
  }

  buildEntryModule(entry) {
    Object.keys(entry).forEach((entryName) = > {
      const entryPath = entry[entryName];
      const entryObj = this.buildModule(entryName, entryPath);
      this.entries.add(entryObj);
    });
  }
  
  
  // Module compilation method
  buildModule(moduleName,modulePath) {
    // ...
    return{}}}Copy the code

Here we have added a method called buildEntryModule as the entry module compilation method. Loop through the entry object to get the name and path of each entry object.

For example, if we pass entry at the beginning :{main:’./ SRC /main.js’}, buildEntryModule gets entry as {main: “/ SRC… [your absolute path]”}, where we’ll take entryName as main and entryPath as the absolute path to the entry file main.

After compiling a single entry, we return an object in the buildModule method. This object is what we compiled the entry file into.

buildModuleModule compilation method

Before we start coding, let’s take a look at what the buildModule method does:

  • BuildModule takes two arguments to build modules, the first being the name of the entry file to which the module belongs and the second being the path to the module to build.

  • The buildModule method starts code compilation by reading the file source from the entry file path through the FS module.

  • After reading the contents of the file, all matching Loaders are called to process the module and the returned result is obtained.

  • After obtaining the result of loader processing, Babel is used to analyze the code processed by Loader and compile the code. (This step is mainly compiled for the require statement, modify the path of the require statement in the source code).

  • If the entry file is not dependent on any module (require statement), the compiled module object is returned.

  • If the entry file has a dependent module, the recursive buildModule method compiles the module.

Reading file contents

  1. Let’s start by callingfsThe module reads the file contents.
const fs = require('fs');
// ...
class Compiler {
      / /...
      // Module compilation method
      buildModule(moduleName, modulePath) {
        // 1. Read the original file code
        const originSourceCode =
          ((this.originSourceCode = fs.readFileSync(modulePath, 'utf-8'));
        // moduleCode is modified code
        this.moduleCode = originSourceCode;
      }
      
      // ...
 }
Copy the code

callloaderProcess files with matching suffixes

  1. Now that we have the contents of the file, we need to match itloaderWe compiled our source code.

Implement a simple custom loader

Before compiling the loader, let’s implement the custom loader we passed in above.

Create loader-1.js,loader-2.js in webpack/loader directory:

Loader is essentially a function that takes our source code as an input parameter and returns the processed result.

You can see more details about loader’s features here. Because the article mainly describes the packaging process, loader is simply processed as reverse order. More specific loader/plugin development will be covered in a future article.

// Loader is essentially a function that takes the original content and returns the converted content.
function loader1(sourceCode) {
  console.log('join loader1');
  return sourceCode + `\n const loader1 = 'https://github.com/19Qingfeng'`;
}

module.exports = loader1;
Copy the code
function loader2(sourceCode) {
  console.log('join loader2');
  return sourceCode + `\n const loader2 = '19Qingfeng'`;
}

module.exports = loader2;
Copy the code

Use loader to process files

After making it clear that loader is a pure function, let’s give the content to the matching Loader before module analysis.

// Module compilation method
  buildModule(moduleName, modulePath) {
    // 1. Read the original file code
    const originSourceCode =
      ((this.originSourceCode = fs.readFileSync(modulePath)), 'utf-8');
    // moduleCode is modified code
    this.moduleCode = originSourceCode;
    // 2. Invoke loader for processing
    this.handleLoader(modulePath);
  }

  // Matches loader processing
  handleLoader(modulePath) {
    const matchLoaders = [];
    // 1. Obtain all incoming Loader rules
    const rules = this.options.module.rules;
    rules.forEach((loader) = > {
      const testRule = loader.test;
      if (testRule.test(modulePath)) {
        if (loader.loader) {
          / / only consider loader {test: / \. Js $/ g, use: [' Babel - loader]}, {test: / \. Js $/, loader, 'Babel - loader'}
          matchLoaders.push(loader.loader);
        } else {
          matchLoaders.push(...loader.use);
        }
      }
      // 2. Execute the loader source code in reverse order
      for (let i = matchLoaders.length - 1; i >= 0; i--) {
        // Currently we only support loader mode passing absolute path
        // require Imports the corresponding loader
        const loaderFn = require(matchLoaders[i]);
        // Synchronously handle my moduleCode every time I compile it through the loader
        this.moduleCode = loaderFn(this.moduleCode); }}); }Copy the code

Here we use the handleLoader function to execute the loader in reverse order to process our code this.moduleCode and synchronously update each moduleCode after the file path is matched to the loader with the corresponding suffix.

Finally, this.Modulecode is processed by the corresponding loader in each module compilation.

webpackModule compilation stage

In the previous step, we went through the loader processing our entry file code, and the processed code is stored in this.Modulecode.

At this point, after loader processing we are about to enter the compilation phase within webpack.

What we need to do here is compile for the current module and change the path introduced by all modules (require()) statements that the current module depends on to the path relative to the path (this.rootPath).

Anyway, what you need to understand is that the result of our compilation here is to expect the dependency module path in the source code to be relative to the path, while establishing the underlying module dependencies. I’ll tell you why I compiled against paths later.

Let’s continue to refine the buildModule method:

const parser = require('@babel/parser');
const traverse = require('@babel/traverse').default;
const generator = require('@babel/generator').default;
const t = require('@babel/types');
const tryExtensions = require('./utils/index')
// ...
  class Compiler {
     // ...
      
     // Module compilation method
      buildModule(moduleName, modulePath) {
        // 1. Read the original file code
        const originSourceCode =
          ((this.originSourceCode = fs.readFileSync(modulePath)), 'utf-8');
        // moduleCode is modified code
        this.moduleCode = originSourceCode;
        // 2. Invoke loader for processing
        this.handleLoader(modulePath);
        // 3. Call Webpack to compile the module to get the final Module object
        const module = this.handleWebpackCompiler(moduleName, modulePath);
        // 4. Return the corresponding module
        return module
      }

      // Call Webpack for module compilation
      handleWebpackCompiler(moduleName, modulePath) {
        // Calculate the relative path of the current module relative to the project startup root as the module ID
        const moduleId = '/' + path.posix.relative(this.rootPath, modulePath);
        // Create a module object
        const module = {
          id: moduleId,
          dependencies: new Set(), // The absolute path address of the module on which the module depends
          name: [moduleName], // The entry file to which the module belongs
        };
        // Call Babel to parse our code
        const ast = parser.parse(this.moduleCode, {
          sourceType: 'module'});// depth-first traverses the syntax Tree
        traverse(ast, {
          // when the require statement is encountered
          CallExpression:(nodePath) = > {
            const node = nodePath.node;
            if (node.callee.name === 'require') {
              // Get the relative path of imported modules in source code
              const moduleName = node.arguments[0].value;
              The current module path +require() corresponds to the relative path
              const moduleDirName = path.posix.dirname(modulePath);
              const absolutePath = tryExtensions(
                path.posix.join(moduleDirName, moduleName),
                this.options.resolve.extensions,
                moduleName,
                moduleDirName
              );
              // generate moduleId - add a new dependent module path to the moduleId corresponding to the path
              const moduleId =
                '/' + path.posix.relative(this.rootPath, absolutePath);
              // Use Babel to change the require in the source code to the __webpack_require__ statement
              node.callee = t.identifier('__webpack_require__');
              // All modules introduced by the require statement in the source code are changed to be handled relative to the path
              node.arguments = [t.stringLiteral(moduleId)];
              // Add the dependency from the require statement to the current module (the module ID relative to the root path)
              module.dependencies.add(moduleId); }}});// Generate new code according to the AST
        const { code } = generator(ast);
        // Mount the newly generated code for the current module
        module._source = code;
        // Returns the current module object
        return module}}Copy the code

This completes our webPack compilation phase.

Note that:

  • Here we compile the require statement using the Babel API. If you are not familiar with the Babel API, you can check out my other two articles in the pre-knowledge section. I’m not a drag here

  • We also refer to a tryExtensions() utility method in the code. This method is for utility methods with incomplete suffixes, which you will see later.

  • For each file compilation, we return a Module object, which is the most important object.

    • idProperty indicating that the current module is targeted atthis.rootPathRelative contents of.
    • dependenciesProperty, it’s aSetThe ids of all modules on which the module depends are stored internally.
    • nameProperty that indicates which entry file the module belongs to.
    • _sourceProperty, which stores the module’s own pathbabelCompiled string code.

TryExtensions method implementation

We had this configuration in webpack.config.js above:

Those of you familiar with WebPack configuration may know that webPack automatically suffixes files that have not been suffixed when importing dependencies.

Now that we know how this works let’s take a look at the implementation of the utils/tryExtensions method:


/ * * * * *@param {*} ModulePath absolute modulePath *@param {*} Extensions Array *@param {*} OriginModulePath Original imported module path *@param {*} ModuleContext moduleContext (current module directory) */
function tryExtensions(modulePath, extensions, originModulePath, moduleContext) {
  // The extension option is not required
  extensions.unshift(' ');
  for (let extension of extensions) {
    if (fs.existsSync(modulePath + extension)) {
      returnmodulePath + extension; }}// The corresponding file is not matched
  throw new Error(
    `No module, Error: Can't resolve ${originModulePath} in  ${moduleContext}`
  );
}
Copy the code

This method is very simple. We check the incoming file with fs.existssync to see if there is a matching path and return it if there is one. If it is not found, a friendly error is given.

Extensions. Unshift (”); To prevent users from passing in suffixes, we first try to find them directly, and then return them if they can find the file. If you can’t find one, you try one at a time.

Recursive processing

After the previous step, we can call buildModule for the entry file to get this return object.

Let’s take a look at the result of running webpack/core/index.js.

I printed the processed entries object in buildEntryModule. You can see what we expected earlier:

  • idFor each module relative to the module with the pathcontext:process.cwd()) forwebpackDirectory.
  • dependenciesIs a dependent module within the module, which has not been added yet.
  • nameIs the name of the entry file to which the module belongs.
  • _sourceCompiled source code for the module.

The current content in _source is based on

Now let’s open the SRC directory and add some dependencies and content to our two entry files:

// webpack/example/entry1.js
const depModule = require('./module');

console.log(depModule, 'dep');
console.log('This is entry 1 ! ');


// webpack/example/entry2.js
const depModule = require('./module');

console.log(depModule, 'dep');
console.log('This is entry 2 ! ');

// webpack/example/module.js
const name = '19Qingfeng';

module.exports = {
  name,
};
Copy the code

At this point let’s rerun webpack/core/index.js:

OK, so far we can finish compiling for Entry for now.

In short, this step we will use the methodentryAnalyze and compile to get an object. Add this object tothis.entries.

Let’s deal with the dependent modules.

The same steps apply to dependent modules:

  • Check for dependencies in the entry file.
  • Call recursively if there are dependenciesbuildModuleMethod compiles the module. The incomingmoduleNameIs the entry file to which the current module belongs.modulePathIs the absolute path of the currently dependent module.
  • In the same way, recursion checks to see if there are still dependencies inside the dependent module. If there are dependencies, recursion compiles the module. This is a depth-first process.
  • Save each compiled module to enterthis.modules.

Now we just need to make a slight change in the handleWebpackCompiler method:

 // Call Webpack for module compilation
  handleWebpackCompiler(moduleName, modulePath) {
    // Calculate the relative path of the current module relative to the project startup root as the module ID
    const moduleId = '/' + path.posix.relative(this.rootPath, modulePath);
    // Create a module object
    const module = {
      id: moduleId,
      dependencies: new Set(), // The absolute path address of the module on which the module depends
      name: [moduleName], // The entry file to which the module belongs
    };
    // Call Babel to parse our code
    const ast = parser.parse(this.moduleCode, {
      sourceType: 'module'});// depth-first traverses the syntax Tree
    traverse(ast, {
      // when the require statement is encountered
      CallExpression: (nodePath) = > {
        const node = nodePath.node;
        if (node.callee.name === 'require') {
          // Get the relative path of imported modules in source code
          const moduleName = node.arguments[0].value;
          The current module path +require() corresponds to the relative path
          const moduleDirName = path.posix.dirname(modulePath);
          const absolutePath = tryExtensions(
            path.posix.join(moduleDirName, moduleName),
            this.options.resolve.extensions,
            moduleName,
            moduleDirName
          );
          // generate moduleId - add a new dependent module path to the moduleId corresponding to the path
          const moduleId =
            '/' + path.posix.relative(this.rootPath, absolutePath);
          // Use Babel to change the require in the source code to the __webpack_require__ statement
          node.callee = t.identifier('__webpack_require__');
          // All modules introduced by the require statement in the source code are changed to be handled relative to the path
          node.arguments = [t.stringLiteral(moduleId)];
          // Add the dependency from the require statement to the current module (the module ID relative to the root path)
          module.dependencies.add(moduleId); }}});// Generate new code according to the AST
    const { code } = generator(ast);
    // Mount the newly generated code for the current module
    module._source = code;
    // If there is a dependency module in recursive dependency depth traversal, add it
    module.dependencies.forEach((dependency) = > {
      const depModule = this.buildModule(moduleName, dependency);
      // Add any compiled dependent module objects to modules objects
      this.modules.add(depModule);
    });
    // Returns the current module object
    return module;
  }
Copy the code

Here we add this code:

    // If there is a dependency module in recursive dependency depth traversal, add it
    module.dependencies.forEach((dependency) = > {
      const depModule = this.buildModule(moduleName, dependency);
      // Add any compiled dependent module objects to modules objects
      this.modules.add(depModule);
    });
Copy the code

Here we make a recursive call to buildModule for the dependent modules, adding the output module object to this.modules.

Now let’s re-run webpack/core/index.js to compile. Here I print assets and modules in buildEntryModule after compiling:

Set{{id: './example/src/entry1.js'.dependencies: Set { './example/src/module.js' },
    name: [ 'main']._source: 'const depModule = __webpack_require__("./example/src/module.js"); \n' +
      '\n' +
      "console.log(depModule, 'dep'); \n" +
      "console.log('This is entry 1 ! '); \n" +
      "const loader2 = '19Qingfeng'; \n" +
      "const loader1 = 'https://github.com/19Qingfeng';"
  },
  {
    id: './example/src/entry2.js'.dependencies: Set { './example/src/module.js' },
    name: [ 'second']._source: 'const depModule = __webpack_require__("./example/src/module.js"); \n' +
      '\n' +
      "console.log(depModule, 'dep'); \n" +
      "console.log('This is entry 2 ! '); \n" +
      "const loader2 = '19Qingfeng'; \n" +
      "const loader1 = 'https://github.com/19Qingfeng';"
  }
} entries
Set{{id: './example/src/module.js'.dependencies: Set {},
    name: [ 'main']._source: "const name = '19Qingfeng'; \n" +
      'module.exports = {\n' +
      ' name\n' +
      '}; \n' +
      "const loader2 = '19Qingfeng'; \n" +
      "const loader1 = 'https://github.com/19Qingfeng';"
  },
  {
    id: './example/src/module.js'.dependencies: Set {},
    name: [ 'second']._source: "const name = '19Qingfeng'; \n" +
      'module.exports = {\n' +
      ' name\n' +
      '}; \n' +
      "const loader2 = '19Qingfeng'; \n" +
      "const loader1 = 'https://github.com/19Qingfeng';"
  }
} modules
Copy the code

You can see that we’ve added the module.js dependency to modules as expected, and it’s also handled by the Loader. But we found that it had been added twice.

This is because the module.js module is referenced twice, it is already dependent on entry1 and entry2, and we buildModule the same module twice during recursive compilation.

Let’s deal with this:

    handleWebpackCompiler(moduleName, modulePath){...// Use Babel to change the require in the source code to the __webpack_require__ statement
          node.callee = t.identifier('__webpack_require__');
          // All modules introduced by the require statement in the source code are changed to be handled relative to the path
          node.arguments = [t.stringLiteral(moduleId)];
          // The array converted to IDS is easy to handle
          const alreadyModules = Array.from(this.modules).map((i) = > i.id);
          if(! alreadyModules.includes(moduleId)) {// Add the dependency from the require statement to the current module (the module ID relative to the root path)
            module.dependencies.add(moduleId);
          } else {
            // If it already exists, do not add the entry to the module compilation but still update the entry that the module depends on
            this.modules.forEach((value) = > {
              if(value.id === moduleId) { value.name.push(moduleName); }}); }}}}); . }Copy the code

Here, in the dependency conversion of each code analysis, we first determine whether the this.module object already has the current module (as determined by the unique module ID path).

If it doesn’t exist, add it to the dependency for compilation. If the module already exists, the module is already compiled. Therefore, we do not need to compile it again at this time, we just need to update the chunk that the module belongs to and add the current chunk name to its name attribute.

Rerun and let’s look at the print:

Set{{id: './example/src/entry1.js'.dependencies: Set { './example/src/module.js' },
    name: [ 'main']._source: 'const depModule = __webpack_require__("./example/src/module.js"); \n' +
      '\n' +
      "console.log(depModule, 'dep'); \n" +
      "console.log('This is entry 1 ! '); \n" +
      "const loader2 = '19Qingfeng'; \n" +
      "const loader1 = 'https://github.com/19Qingfeng';"
  },
  {
    id: './example/src/entry2.js'.dependencies: Set {},
    name: [ 'second']._source: 'const depModule = __webpack_require__("./example/src/module.js"); \n' +
      '\n' +
      "console.log(depModule, 'dep'); \n" +
      "console.log('This is entry 2 ! '); \n" +
      "const loader2 = '19Qingfeng'; \n" +
      "const loader1 = 'https://github.com/19Qingfeng';"
  }
} entries
Set{{id: './example/src/module.js'.dependencies: Set {},
    name: [ 'main'.'./module']._source: "const name = '19Qingfeng'; \n" +
      'module.exports = {\n' +
      ' name\n' +
      '}; \n' +
      "const loader2 = '19Qingfeng'; \n" +
      "const loader1 = 'https://github.com/19Qingfeng';"
  }
} modules
Copy the code

At this point, the “module compilation stage” for us is basically over. In this step, we analyze all modules from the entry file.

  • Starting from the entry, reading the contents of the entry file calls the matchloaderProcess import files.
  • throughbabelAnalyze the dependencies and change all dependent paths relative to the project startup directory at the same timeoptions.contextThe path.
  • Recursively compile the dependency module if there are dependencies in the entry file.
  • Add compiled objects for each dependent modulethis.modules.
  • Add the compiled objects for each entry filethis.entries.

Compile completion phase

In the previous step, we finished compiling between modules and filled in the contents for module and entry respectively.

After compiling all modules recursively, we need to combine the final output chunk modules according to the above dependencies.

Let’s continue with our Compiler:

class Compiler {

    // ...
    buildEntryModule(entry) {
        Object.keys(entry).forEach((entryName) = > {
          const entryPath = entry[entryName];
          // Call buildModule to implement the real module compilation logic
          const entryObj = this.buildModule(entryName, entryPath);
          this.entries.add(entryObj);
          // According to the interdependence between the file and module of the current entry, assemble into a chunk containing all the dependent modules of the current entry
          this.buildUpChunk(entryName, entryObj);
        });
        console.log(this.chunks, 'chunks');
    }
    
     // Assemble the chunks according to the entry file and dependency modules
      buildUpChunk(entryName, entryObj) {
        const chunk = {
          name: entryName, // Each entry file is a chunk
          entryModule: entryObj, // entry Specifies the compiled object
          modules: Array.from(this.modules).filter((i) = >
            i.name.includes(entryName)
          ), // Find all modules associated with the current entry
        };
        // add chunks to this.chunks
        this.chunks.add(chunk);
      }
      
      // ...
}
Copy the code

Here, we find all dependent files of each module based on the corresponding entry file through the name attribute of each module.

Let’s look at what this.chunks will eventually output:

Set{{name: 'main'.entryModule: {
      id: './example/src/entry1.js'.dependencies: [Set].name: [Array]._source: 'const depModule = __webpack_require__("./example/src/module.js"); \n' +
        '\n' +
        "console.log(depModule, 'dep'); \n" +
        "console.log('This is entry 1 ! '); \n" +
        "const loader2 = '19Qingfeng'; \n" +
        "const loader1 = 'https://github.com/19Qingfeng';"
    },
    modules: [[Object]]}, {name: 'second'.entryModule: {
      id: './example/src/entry2.js'.dependencies: Set {},
      name: [Array]._source: 'const depModule = __webpack_require__("./example/src/module.js"); \n' +
        '\n' +
        "console.log(depModule, 'dep'); \n" +
        "console.log('This is entry 2 ! '); \n" +
        "const loader2 = '19Qingfeng'; \n" +
        "const loader1 = 'https://github.com/19Qingfeng';"
    },
    modules: []}}Copy the code

In this step, we get two chunks of the final output from the Webpack.

They have:

  • name: Specifies the name of the current entry file
  • entryModule: entry file compiled object.
  • modules: The entry file depends on an array of all module objects, the format of each element andentryModuleIt’s consistent.

At this point the compilation is complete and our chunk assembly is complete.

Output file stage

Let’s start with this.chunks from the previous compilation.

Analyze the raw packaging output

Here I have made the following changes to webpack/core/index.js:

- const webpack = require('./webpack');
+ const webpack = require('webpack')...Copy the code

Use the original Webpack instead of our own implementation of the Webpack first for a package.

After running webpack/core/index.js, we get two files in webpack/ SRC /build :main.js and second.js. Let’s look at the contents of one of them:

(() = > {
  var __webpack_modules__ = {
    './example/src/module.js': (module) = > {
      const name = '19Qingfeng';

      module.exports = {
        name,
      };

      const loader2 = '19Qingfeng';
      const loader1 = 'https://github.com/19Qingfeng'; }};// The module cache
  var __webpack_module_cache__ = {};

  // The require function
  function __webpack_require__(moduleId) {
    // Check if module is in cache
    var cachedModule = __webpack_module_cache__[moduleId];
    if(cachedModule ! = =undefined) {
      return cachedModule.exports;
    }
    // Create a new module (and put it into the cache)
    var module = (__webpack_module_cache__[moduleId] = {
      // no module.id needed
      // no module.loaded needed
      exports: {},});// Execute the module function
    __webpack_modules__[moduleId](module.module.exports, __webpack_require__);

    // Return the exports of the module
    return module.exports;
  }

  var __webpack_exports__ = {};
  // This entry need to be wrapped in an IIFE because it need to be isolated against other modules in the chunk.
  (() = > {
    const depModule = __webpack_require__(
      / *! ./module */ './example/src/module.js'
    );

    console.log(depModule, 'dep');
    console.log('This is entry 1 ! ');

    const loader2 = '19Qingfeng';
    const loader1 = 'https://github.com/19Qingfeng'; }) (); }) ();Copy the code

Here, I manually removed the extra comments generated after the package and streamlined the code.

Let’s take a look at the code generated by the original package:

The webpack code internally defines a __webpack_require__ function in place of the NodeJs require method.

And at the bottom

This piece of code is familiar to you, and this is the entry file code that we compiled. And the code at the top is an object defined by all modules that the entry file depends on:

Here we define a __webpack__modules object. The key of the ** object is the relative path of the dependent module to the path, and the value of the object is the compiled code of the dependent module. `

Output file stage

Next, after analyzing the original webPack code, let’s continue with the previous step. Try this with our this.chunks.

Let’s go back to the run method on Compiler:

   class Compiler {}// The run method starts compilation
  // Meanwhile, the run method accepts an externally passed callback
  run(callback) {
    // The plugin that starts compiling is triggered when the run mode is called
    this.hooks.run.call();
    // Get the entry configuration object
    const entry = this.getEntry();
    // Compile the entry file
    this.buildEntryModule(entry);
    // Export list; Then convert each chunk into a separate file and add it to the output list assets
    this.exportFile(callback);
  }
Copy the code

After compiling the buildEntryModule module, we implement the logic of exporting the file through this. ExportFile method.

Let’s take a look at the this.exportFile method:

 // Add chunk to the output list
  exportFile(callback) {
    const output = this.options.output;
    // Generate assets based on chunks
    this.chunks.forEach((chunk) = > {
      const parseFileName = output.filename.replace('[name]', chunk.name);
      // Assets {'main.js': 'Generated string code... '}
      this.assets.set(parseFileName, getSourceCode(chunk));
    });
    // call the Plugin emit hook
    this.hooks.emit.call();
    If fs.write does not exist, create the directory first
    if(! fs.existsSync(output.path)) { fs.mkdirSync(output.path); }// files to save all generated file names
    this.files = Object.keys(this.assets);
    // Write the contents of assets to the file system
    Object.keys(this.assets).forEach((fileName) = > {
      const filePath = path.join(output.path, fileName);
      fs.writeFileSync(filePath, this.assets[fileName]);
    });
    // Trigger the hook when finished
    this.hooks.done.call();
    callback(null, {
      toJson: () = > {
        return {
          entries: this.entries,
          modules: this.modules,
          files: this.files,
          chunks: this.chunks,
          assets: this.assets, }; }}); }Copy the code

ExportFile does several things:

  • We first get the output configuration of the configuration parameters, iterate through our this.chunks, and replace [name] in output.filename with the corresponding entry filename. At the same time, add the filename and content of this. Assets to be packaged according to the content of the chunks.

  • Call plugin’s emit hook function before writing the file to disk.

  • Check whether the output.path folder exists. If not, create a new folder through fs.

  • Put all file names (array of key values of this.assets) generated in this package into files.

  • Loop through this.assets to write the files in turn to the corresponding disk.

  • The done hook of the WebPack plug-in is triggered when the packaging process is complete.

  • In response to the NodeJs Webpack APi, two arguments are passed to the callback passed externally in the run method.

In general, what this.assets does is fairly simple: it analyzes chunks to get assets and then outputs the corresponding code to disk.

If you look at the code above, you’ll see. The value of each element in this. Assets Map is generated by calling getSourceCode(chunk) to generate the corresponding module code.

So how does the getSourceCode method generate our final compiled code based on chunk? Let’s take a look!

getSourceCodemethods

First, let’s briefly clarify what this method does. We need the getSourceCode method to accept the incoming chunk object. This returns the chunk’s source code.

Without further ado, in fact, I used a lazy method here, but it does not prevent you from understanding the Webpack process. We have analyzed the original Webpack code, only the entry file and module dependencies are packaged in different places each time, and the require method and so on are the same.

To grasp the difference each time, let’s get straight to the implementation:

// webpack/utils/index.js./ * * * * *@param {*} chunk* name Attribute entry file name * entryModule Entry file Module object * modules dependency module path */
function getSourceCode(chunk) {
  const { name, entryModule, modules } = chunk;
  return `
  (() => {
    var __webpack_modules__ = {
      ${modules
        .map((module) = > {return `
          'The ${module.id}': (module) => {
            The ${module._source}
      }
        `;
        })
        .join(', ')}}; // The module cache var __webpack_module_cache__ = {}; // The require function function __webpack_require__(moduleId) { // Check if module is in cache var cachedModule = __webpack_module_cache__[moduleId]; if (cachedModule ! == undefined) { return cachedModule.exports; } // Create a new module (and put it into the cache) var module = (__webpack_module_cache__[moduleId] = { // no module.id needed // no module.loaded needed exports: {}, }); // Execute the module function __webpack_modules__[moduleId](module, module.exports, __webpack_require__); // Return the exports of the module return module.exports; } var __webpack_exports__ = {}; // This entry need to be wrapped in an IIFE because it need to be isolated against other modules in the chunk. (() => {${entryModule._source}}) (); }) (); `; }...Copy the code

This code is actually very, very simple, far less difficult than you think! Kind of back to basics, huh? Ha ha.

In the getSourceCode method, we get the corresponding chunk from the combined chunk:

  • name: The name of the import file corresponding to the output file.
  • entryModule: stores the compiled object of the entry file.
  • modules: The object that holds all modules that the entry file depends on.

We do that by concatenating strings__webpack__modulesProperties on the object, also pass at the bottom${entryModule._source}Splice the code of the inbound and outbound files.

Here we mentioned why we need to convert the module’s require method path to a path relative to the context, and I’m sure we already know why. Because the __webpack_require__ methods we end up implementing are all self-implemented require methods for modules relative to paths.

Also, if you’re not sure how the require method is converted to the __webpack_require__ method, you can go back to our compilation section. We changed the require method call to __webpack_require__ during the AST conversion phase with Babel.

You’re done

At this point, let’s go back to webpack/core/index.js. Re-run the file and you’ll see a build directory in the webpack/example directory.

In this step we have perfectly implemented our own WebPack.

In essence, we want to implement a simple version of the WebPack core with a thorough understanding of the Compiler object as well as its workflow.

In any subsequent development related to Webpack, the use of compiler is really clear. Learn how various properties on the Compiler affect the result of compiling and packaging.

Let’s finish with a flow chart:

Write in the last

First of all, thank you to everyone who can see here.

This article has a certain knowledge threshold and a large number of code parts, I admire everyone who can read to the end of the students.

This is the end of the article for implementing a simple version of Webpack, which is really just a basic version of the Webpack workflow.

But it is through such a small 🌰 can take us to really start the core workflow of Webpack, I hope this article can play a better auxiliary role for everyone to understand Webpack.

In fact, after a clear understanding of the basic workflow, the development of Loader and Plugin is easy to do. The development introduction of these two parts in this article is superficial. I will update the detailed development process of Loader and Plugin respectively in the future. Interested students can follow 😄.

The code in this article can be downloaded here, and I will continue to refine more workflow logic in the code base for this easy version of Webpack.

At the same time, the code here I want to emphasize is the source code flow explanation, the real Webpack will be much more complex than here. Here for the convenience of everyone to understand deliberately simplified, but the core workflow is basically consistent with the source code.