This is the seventh day of my participation in the August More text Challenge. For details, see: August More Text Challenge

This article was written on and off for several hours, trying to clarify the bundler issue, mainly because the bundle issue was trivial and complicated. I’m not sure if I’ve made it clear yet, but technical documentation is inherently rigorous because of its inherent logic. Even so, it is not easy to express it properly, it must be even harder to tell a story clearly, and suddenly I feel a sense of awe for literary writers.

First of all, we all know today for any language, not package management tools (modules), because for large applications, is split to code according to the function modular, the benefits of this modular is self-evident, will only see the modularity benefits in the development process, modular can let everybody together points module to parallel development.

Javascript started out without the concept of modules. In the early days, javascript was mainly used to add dynamic effects and interactive toys to web pages. It was probably not necessary to introduce the concept of modules, but today, as Web applications become more complex and more things are done on the front end, And with nodeJS expanding the domain of javascript from the front end to the back end, the introduction of package management into the language is imminent. The first javascript package I touched had require.js. Today, there are two main concepts of javascript package management tools that everyone implements.

  • ES6 module
import _ from 'lodash'

export default someValue;
Copy the code

It is important to note that the ES Module form needs to start the service, which is valued through HTTP requests

  • Commonjs modules
const _ = require('lodash');
module.exports = someValue
Copy the code

motivation

Webpack is a great javascript package management tool. Many projects are built with Webpack, and it seems that the Vue project recently abandoned Webpack in favor of the rollup.js build tool. I have been exposed to the early require.js and used WebPack to build projects, but only at the level of use, the specific internal mechanism is not yet understood. Yesterday saw a foreigner to share on how to achieve a javascript building tool video, ready to implement a with you, of course, is not only copy, to share it will be on some key content to expand, and is a local extension.

Train of thought

To do one thing, you need to think about how to do it first, for example, put the elephant in the refrigerator, which is divided into three parts, so we should do the project build how to do, think about, first we need to read the JS file, and then find the information from the file that contains the dependency,

  • Read js to parse the file into an AST
  • The code is abstracted from the AST to extract dependency information
  • Dependency graphs that can be traversed using correlation information are generated
  • Use dependency diagrams to organize code efficiently

entry.js - message.js - name.js
Copy the code

Obtaining resource Objects

We treat each javascript file as a resource, and the createAsset method here is mainly responsible for reading each javascript file (as a resource) and extracting the required information from the resource.

Reading javascript files

The fs module is introduced, and the file content is read by readFileSync.

const fs = require("fs")
Copy the code
function createAsset(filename){
    const content = fs.readFileSync(filename,'utf-8');

    console.log(content);
}

creatAsset('./example/entry.js');
Copy the code

Parse the text into an AST

Here createAsset takes the resource, reads the file as a string and assigns it to the Content object. Then we need to read the string to parse, parse into an AST, then what is an AST, the full name is abstract syntax tree, that is, javascript code to form a structured syntax tree, in fact, AST is a common Json file, used to describe.

Here we introduce the Babylon package to help us parse the string into an AST structure data. Next we need to parse the file to find out which files the file depends on,

  • File Program body ImportDeclaration
  • Every node has a type
const fs = require("fs");
const babylon = require('babylon');

function creatAsset(filename){
    const content = fs.readFileSync(filename,'utf-8');

    const ast = babylon.parse(content,{
        sourceType:'module'
    });

    console.log(ast);
}
Copy the code

The following is our parsing output AST object, which is a normal JSON object that we can read briefly. Each node has a type. Let’s take a quick look at what each node contains. Type indicates the type of the node, and start and end indicate where the node is in code.

import message from './message.js';

console.log(message)
Copy the code

There are two sets of comments and tokens, and one is stored in the tokens

Node {
  type: 'File'.start: 0.end: 57.loc: SourceLocation {
    start: Position { line: 1.column: 0 },
    end: Position { line: 3.column: 20}},program: Node {
    type: 'Program'.start: 0.end: 57.loc: SourceLocation { start: [Position], end: [Position] },
    sourceType: 'module'.body: [ [Node], [Node] ],
    directives: []},comments: [].tokens: [
    Token {
      type: [KeywordTokenType],
      value: 'import'.start: 0.end: 6.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: 'message'.start: 7.end: 14.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: 'from'.start: 15.end: 19.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: './message.js'.start: 20.end: 34.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: undefined.start: 34.end: 35.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: 'console'.start: 37.end: 44.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: undefined.start: 44.end: 45.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: 'log'.start: 45.end: 48.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: undefined.start: 48.end: 49.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: 'message'.start: 49.end: 56.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: undefined.start: 56.end: 57.loc: [SourceLocation]
    },
    Token {
      type: [TokenType],
      value: undefined.start: 57.end: 57.loc: [SourceLocation]
    }
  ]
}
Copy the code

The next step is to refine the createAsset. After we have parsed the text to the AST, we introduce the Babel-traverse to get the ImportDeclaration on the AST that is parsed.

/ * * * /

const fs = require("fs");
const babylon = require('babylon');
const traverse = require('babel-traverse').default;

function creatAsset(filename){
    const content = fs.readFileSync(filename,'utf-8');

    const ast = babylon.parse(content,{
        sourceType:'module'
    });

    traverse(ast,{
        ImportDeclaration:({node}) = >{
            console.log(node); }})// console.log(ast);
}

creatAsset('./example/entry.js');
Copy the code

The import content is extracted from the traverse, and the node source.value is used to get./message.js which is what we want, and the ImportDeclaration node is used to get the dependencies of the file.

Node {
  type: 'ImportDeclaration'.start: 0.end: 35.loc: SourceLocation {
    start: Position { line: 1.column: 0 },
    end: Position { line: 1.column: 35}},specifiers: [
    Node {
      type: 'ImportDefaultSpecifier'.start: 7.end: 14.loc: [SourceLocation],
      local: [Node]
    }
  ],
  source: Node {
    type: 'StringLiteral'.start: 20.end: 34.loc: SourceLocation { start: [Position], end: [Position] },
    extra: { rawValue: './message.js'.raw: "'./message.js'" },
    value: './message.js'}}Copy the code

The dependencies collected are stored in the Dependencies array.

function creatAsset(filename){
    const content = fs.readFileSync(filename,'utf-8');

    const ast = babylon.parse(content,{
        sourceType:'module'
    });

    const dependencies = []

    traverse(ast,{
        ImportDeclaration:({node}) = >{
            // console.log(node);dependencies.push(node.source.value); }});console.log(dependencies);
}
Copy the code
[ './message.js' ]
Copy the code

We createAsset returns a resource object with attributes such as ID, filename, and dependencies, which hold the files that the file depends on.

let ID = 0;

function createAsset(filename){
    const content = fs.readFileSync(filename,'utf-8');

    const ast = babylon.parse(content,{
        sourceType:'module'
    });

    const dependencies = []

    traverse(ast,{
        ImportDeclaration:({node}) = >{
            // console.log(node);dependencies.push(node.source.value); }});const id = ID++;
    return{
        id,
        filename,
        dependencies,
    };
    // console.log(dependencies);
}

const mainAsset = creatAsset('./example/entry.js');
console.log(mainAsset)
Copy the code
{
  id: 0.filename: './example/entry.js'.dependencies: [ './message.js']}Copy the code

Creating a dependency graph

To build the file dependency graph, first place the incoming resource in a queue and walk through the dependencies, which contain the path of the file dependency. In this case, the dependency path is relative, but createAsset needs to accept an absolute path as an argument. Import message from ‘./message.js’; import message from ‘./message.js’; .

function createGraph(entry){
    const mainAsset = createAsset(entry);

    const queue = [mainAsset];
    for(const asset of queue){
        const dirname = path.dirname(asset.filename);

        asset.mapping = {}

        asset.dependencies.forEach(relativePath= > {
            const absolutePath = path.join(dirname,relativePath);
            const child = createAsset(absolutePath);
						asset.mapping[relativePath] = child.id
            queue.push(child)
        });
    }
    return queue;
}
Copy the code

Create a graph that represents the dependencies of each file in the project. Create a createGraph() method that describes the path of the main entry file as an argument. Then call createAsset to retrieve the id,filename, and dependencies of the object. I’m going to define a queue, which is going to hold all of the, let’s call it resource objects, and every time we get the resource object, we’re going to get the dependency file path from the dependency array in the resource object, because the dependency file path is relative to the path that references the file, When creatAsset reads a file, it requires an absolute path. So introduce the path package by const absolutePath = path.join(dirname,relativePath); You get the absolute path.

[{id: 0.filename: './example/entry.js'.dependencies: [ './message.js'].mapping: { './message.js': 1}}, {id: 1.filename: 'example/message.js'.dependencies: [ './name.js'].mapping: { './name.js': 2}}, {id: 2.filename: 'example/name.js'.dependencies: [].mapping: {}}]Copy the code

Bundle based on dependency graphs

A bundle is a group of module files based on a dependency graph,

function bundle(graph){

    let modules = ' ';
    
    graph.forEach(mod= > {
        modules + `${mod.i}` : []
    })

    const result = `
        (function(){

        })({${modules}`})}
Copy the code
const result = `
        (function(){

        })({${modules}`})}
Copy the code

Return a result function that executes immediately, passing an object as an argument.

Since not all browsers use EMSCscript module form, and most support commonJs as a package reference, Bebel can be seen as a code conversion tool that converts code to other forms depending on the presets you give :[‘env’].

    const {code} = babel.transformFromAst(ast,null, {presets: ['env']});
    // install babel-core and npm install babel-preset-env --save

    return{
        id,
        filename,
        dependencies,
        code
    };
Copy the code

Next, add a code property and a Mapping property to each resource object.

[{id: 0.filename: './example/entry.js'.dependencies: [ './message.js'].code: '"use strict"; \n' +
      '\n' +
      'var _message = require("./message.js"); \n' +
      '\n' +
      'var _message2 = _interopRequireDefault(_message); \n' +
      '\n' +
      'function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }\n' +
      '\n' +
      'console.log(_message2.default); '.mapping: { './message.js': 1}}, {id: 1.filename: 'example/message.js'.dependencies: [ './name.js'].code: '"use strict"; \n' +
      '\n' +
      'Object.defineProperty(exports, "__esModule", {\n' +
      ' value: true\n' +
      '}); \n' +
      '\n' +
      'var _name = require("./name.js"); \n' +
      '\n' +
      'exports.default = "hello " + _name.name; '.mapping: { './name.js': 2}}, {id: 2.filename: 'example/name.js'.dependencies: [].code: '"use strict"; \n' +
      '\n' +
      'Object.defineProperty(exports, "__esModule", {\n' +
      ' value: true\n' +
      '}); \n' +
      "var name = exports.name = 'machine learning';".mapping: {}}]Copy the code

function bundle(graph){

    let modules = ' ';
    
    graph.forEach(mod= > {
        modules +=`${mod.id}:[
            function(require, module, exports){
                ${mod.code}
            }
        ]`
    })

    const result = `
        (function(){

        })({${modules}})
    `
    return result;
}
Copy the code
const result = `
  (function(){

  })({${modules}})
`
Copy the code

“Id :[function],id:[function]…” result: {${moduels}} We then put the parsed string into an object.

 (function()({{})0: [function(require.module.exports){
                "use strict";

            var _message = require("./message.js");

            var _message2 = _interopRequireDefault(_message);

            function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }

            console.log(_message2.default); }].1: [function(require.module.exports){
                "use strict";

Object.defineProperty(exports."__esModule", {
  value: true
});

var _name = require("./name.js");

exports.default = "hello "+ _name.name; }].2: [function(require.module.exports){
                "use strict";

Object.defineProperty(exports."__esModule", {
  value: true
});
var name = exports.name = 'machine learning'; }]})Copy the code

Analysis build file

I’m using recursion, and recursion is a marker for beginners,

  • First module is an object, key is an index from 0, and the value is an array containing a function and an object

  • Each function in modules accepts three require, modules, and exports

    function localRequire(relativePath) {
  return require(mapping[relativePath])
}
Copy the code

In this case, mapping is used. Mapping is an object whose key is the relative path to the module./messge.js and its value is the module ID.

So let’s define a require function that takes an ID and then calls the function and passes in 0. So let’s take a look at what’s going on inside require. First we get fn and mapping from the modules array based on the ID. Mapping This is such a key-value pair. We define a localRequire inside require where we recursively call require based on the id of the relative path that the file depends on and that’s why we defined the mapping in the first place. Define module, call function and pass require, module and module.exports.

(function (modules) {
    function require(id) {
        const [fn, mapping] = modules[id]

        function localRequire(relativePath) {
            return require(mapping[relativePath])
        }

        const module = {
            exports: {}}; fn(localRequire,module.module.exports)

        return module.exports
    }
    require(0); ({})0: [
        function (require.module.exports) {
            "use strict";

            var _message = require("./message.js");

            var _message2 = _interopRequireDefault(_message);

            function _interopRequireDefault(obj) {
                return obj && obj.__esModule ? obj : {
                    default: obj
                };
            }

            console.log(_message2.default);
        },
        {
            "./message.js": 1}].1: [
        function (require.module.exports) {
            "use strict";

            Object.defineProperty(exports."__esModule", {
                value: true
            });

            var _name = require("./name.js");

            exports.default = "hello " + _name.name;
        },
        {
            "./name.js": 2}].2: [
        function (require.module.exports) {
            "use strict";

            Object.defineProperty(exports."__esModule", {
                value: true
            });
            var name = exports.name = 'machine learning'; }, {},})Copy the code