This series will be split into two articles

(1) Mainly describe the performance of the problem and dive into webpack watch system. (2) Solve the problem and fundamentally solve the webpack bug

I recently had an interesting problem with an internal tool

Multiple rebuilds occur

When the WebPack entry file is created dynamically for the first time, it causes dozens of recompilations as dependencies are added to the entry file.

After searching, it was found that the issue record traced by Webpack was Files created right before watching starts make watching go into a loop

The problem can be replicated whether you’re using Webpack-dev-middleware or webpack-watch or webpack-dev-server.

Webpack author @Sokra explains it as follows:

The watching may loop in a unlucky case, but this should not result in a different compilation hash. I. e. the webpack-dev-server doesn’t trigger a update if the hash is equal.

There was a problem, but the key programmer operation hash didn’t change, so the upper echelon used it internally.

But in reality, authors like Webpack-dev-server don’t think so!

Crude solutions

As for not wanting to get to the bottom of this, here’s a solution:

// Webpack startup recompilation fix. Remove when @sokra fixes the bug.
// https://github.com/webpack/webpack/issues/2983
// https://github.com/webpack/watchpack/issues/25
const timefix = 11000;
compiler.plugin('watch-run', (watching, callback) => {
  watching.startTime += timefix;
  callback()
});
compiler.plugin('done', (stats) => {
  stats.startTime -= timefix
})Copy the code

warren

Of course, sticking plaster isn’t the focus of this article, but it’s a good glimpse into the overall Watch mechanic in WebPack.

If you don’t want to see so many code snippets, you can also look at the notes I made when I was teasing out the code logic. The red flow is the invocation link at initialization, and the blue part is the callback link after a file change.

The first thing we can be sure of is, Both webpack cli tools, Webpack-dev-middleware and webpack-dev-server realize the function of Watch through Compiler.prototype.watch. In order to achieve the debugging phase of high performance requirements.

To get a clearer picture of the process, start by creating an instance of Compiler

Creation of an instance of webpack Compiler

Const compiler = webpack(webpackConfig); This is used to create an instance of the Compiler, also known as an instance of webpack. The Compiler instance object contains all parameters related to packaging, plugins loaders, and so on. In this case, WebPack does not do build compilation by default, but instead executes Compiler.run (callback) to initiate compilation. We can also use webpack(webpackConfig, callback); Starts the build build process by default.

As for the watch process we want to know today, we only need to know that the trend of the whole process is Compiler.watch (watchOptions, callback) when the watch configuration item is explicitly enabled in the construction parameter. Rather than a compiler. The run (the callback); .

A side note: Compilation may be interesting to you, but it contains chunks Modules and other information. Compilation is regenerated whenever changes are made to dependencies. Compiler has only one compiler.

Source tracing

Create the Watch service in Compiler. watch

Class Compiler extends Tapable {watch(watchOptions, handler) {... const watching = new Watching(this, watchOptions, handler); return watching; }}Copy the code
// Watch class class Watching {constructor(compiler, watchOptions, handler) {this.startTime = null; . this.compiler = compiler; this.compiler.readRecords(err => { if(err) return this._done(err); this._go(); }); }}Copy the code

Note here that the _go method is called every time startTime is compiled and assigned a compile startTime, which is very, very important in determining whether the file needs to be compiled again or changed!

Source tracing 1 Source tracing 2

Initialization of the first compilation

When this._go() is executed as above, the first compilation process begins

_go() {
    this.startTime = Date.now();
    this.running = true;
    this.invalid = false;
    this.compiler.applyPluginsAsync("watch-run", this, err => {
        if(err) return this._done(err);
        const onCompiled = (err, compilation) => {
          ...
          this.compiler.emitAssets(compilation, err => {
            ...
            return this._done(null, compilation);
          });
        };
        this.compiler.compile(onCompiled);
    });
}Copy the code

Knock on the board: Notice that startTime is officially assigned to the time when the first build starts, and that the execution of compile marks the start of the first build.

This article does not cover the webpack event flow, loaders and plugins, etc., in the compilation process, we just need to know that after compiling, we enter the compilation process.

As can be seen from the code, the _done method is called after the normal compilation process is completed under the normal flow.

_done(err, compilation) { ... const stats = compilation ? this._getStats(compilation) : null; . this.compiler.applyPlugins("done", stats); . if(! this.closed) { this.watch(compilation.fileDependencies, compilation.contextDependencies, compilation.missingDependencies); }}Copy the code

In the compilation object we can get all the dependencies associated with the build, and those dependencies are really what we need to listen for.

Source tracing

File listening is enabled

We saw in the last procedure that we ended up passing the build dependency to the watch method.

watch(files, dirs, missing) {
    this.pausedWatcher = null;
    this.watcher = this.compiler.watchFileSystem.watch(files, dirs, missing, this.startTime, this.watchOptions, (err, filesModified, contextModified, missingModified, fileTimestamps, contextTimestamps) => {
        ...
        this.invalidate();
    }, (fileName, changeTime) => {
        this.compiler.applyPlugins("invalid", fileName, changeTime);
    });
}
Copy the code

We noticed that watch the real call here is that the compiler. WatchFileSystem. Watch. Those who read the source code may be curious, as the Compiler source code does not define methods on the prototype chain. The reason is very simple, because in the Webpack (webpackConfig) stage, Webpack injected a lot of internal own plug-ins, Webpack source code is very worth learning a point is plug-in mechanism application such fire pure feeling. We can see the webpack.js, and from this we can find the NodeEnvironmentPlugin and start to see the familiar watch word NodeWatchFileSystem, This led us to the NodeWatchFileSystem and the watchPack, the eventual starter of the Watch service.

An aside: The interesting thing here is the NodeEnvironmentPlugin plugin, NodeOutputFileSystem NodeJsInputFileSystem CachedInputFileSystem is set by default in this plugin. By default, after the compilation of webpack, the file content will be output to the actual file directory through IO, but after all, the performance involving IO operations cannot meet the needs of debugging. So webpack-dev-middleware boosts performance by replacing the NodeOutputFileSystem’s default FS with memory-fs. The CachedInputFileSystem and others are also physically accelerated by locally built cache files. Since these topics are not the focus of this article, I will not expand on them, and interested students can continue to dig deeper.

const Watchpack = require("watchpack"); class NodeWatchFileSystem { constructor(inputFileSystem) { this.inputFileSystem = inputFileSystem; this.watcherOptions = { aggregateTimeout: 0 }; this.watcher = new Watchpack(this.watcherOptions); } watch(files, dirs, missing, startTime, options, callback, callbackUndelayed) { ... const oldWatcher = this.watcher; this.watcher = new Watchpack(options); . if(callbackUndelayed) this.watcher.once("change", callbackUndelayed); this.watcher.once("aggregated", (changes, removals) => { ... const times = this.watcher.getTimes(); callback(null, changes.filter(file => files.indexOf(file) >= 0).sort(), changes.filter(file => dirs.indexOf(file) >= 0).sort(), changes.filter(file => missing.indexOf(file) >= 0).sort(), times, times); }); . this.watcher.watch(files.concat(missing), dirs.concat(missing), startTime); if(oldWatcher) { oldWatcher.close(); }... }}Copy the code

Based on the source code of Webpack, it is not difficult to find that the final watch is handed over to the Watch method of the Watchpack instance.

And then we see

Watchpack.prototype.watch = function watch(files, directories, startTime) {
    this.paused = false;
    var oldFileWatchers = this.fileWatchers;
    var oldDirWatchers = this.dirWatchers;
    this.fileWatchers = files.map(function(file) {
        return this._fileWatcher(file, watcherManager.watchFile(file, this.watcherOptions, startTime));
    }, this);
    this.dirWatchers = directories.map(function(dir) {
        return this._dirWatcher(dir, watcherManager.watchDirectory(dir, this.watcherOptions, startTime));
    }, this);
    oldFileWatchers.forEach(function(w) {
        w.close();
    }, this);
    oldDirWatchers.forEach(function(w) {
        w.close();
    }, this);
};Copy the code

For those of you who are not familiar with Webpack, you may be confused as to why file and dir should be distinguished from Watch. By default, webpack Resolve allows you to get the exact path address of each module, but for some special uses, For example, require.context(path) is used to listen on the directory corresponding to that path.

So in general business scenarios only this._fileWatcher is involved.

Watchpack.prototype._fileWatcher = function _fileWatcher(file, watcher) {
    watcher.on("change", function(mtime, type) {
        this._onChange(file, mtime, file, type);
    }.bind(this));
    watcher.on("remove", function(type) {
        this._onRemove(file, file, type);
    }.bind(this));
    return watcher;
};Copy the code

Watchermanager. watchFile(file, this.watcherOptions, StartTime returns a watcher and _fileWather basically does an event binding on the returned watcher.

Watchermanager. watchFile(file, this.watcherOptions, startTime)

WatcherManager.prototype.getDirectoryWatcher = function(directory, options) { var DirectoryWatcher = require("./DirectoryWatcher"); options = options || {}; var key = directory + " " + JSON.stringify(options); if(! this.directoryWatchers[key]) { this.directoryWatchers[key] = new DirectoryWatcher(directory, options); this.directoryWatchers[key].on("closed", function() { delete this.directoryWatchers[key]; }.bind(this)); } return this.directoryWatchers[key]; }; WatcherManager.prototype.watchFile = function watchFile(p, options, startTime) { var directory = path.dirname(p); return this.getDirectoryWatcher(directory, options).watch(p, startTime); }; WatcherManager.prototype.watchDirectory = function watchDirectory(directory, options, startTime) { return this.getDirectoryWatcher(directory, options).watch(directory, startTime); };Copy the code

Step1: this.getDirectoryWatcher(directory, options) : getDirectoryWatcher(directory, options

The implication is that all files in a directory will be mapped to a directoryWatcher.

When creating a new instance of DirectoryWatcher

function DirectoryWatcher(directoryPath, options) {
    EventEmitter.call(this);
    this.options = options;
    this.path = directoryPath;
    this.files = Object.create(null);
    this.directories = Object.create(null);
    this.watcher = chokidar.watch(directoryPath, {
        ignoreInitial: true,
        persistent: true,
        followSymlinks: false,
        depth: 0,
        atomic: false,
        alwaysStat: true,
        ignorePermissionErrors: true,
        ignored: options.ignored,
        usePolling: options.poll ? true : undefined,
        interval: typeof options.poll === "number" ? options.poll : undefined
    });
    this.watcher.on("add", this.onFileAdded.bind(this));
    this.watcher.on("addDir", this.onDirectoryAdded.bind(this));
    this.watcher.on("change", this.onChange.bind(this));
    this.watcher.on("unlink", this.onFileUnlinked.bind(this));
    this.watcher.on("unlinkDir", this.onDirectoryUnlinked.bind(this));
    this.watcher.on("error", this.onWatcherError.bind(this));
    this.initialScan = true;
    this.nestedWatching = false;
    this.initialScanRemoved = [];
    this.doInitialScan();
    this.watchers = Object.create(null);
}Copy the code

You can see that the webpack Watch folder changes are actually exported by Chokidar and bind add, addDir, change, unlink to directoryPath’s Chokidar watcher. Events such as unlinkDir and error. This.doinitialscan () is executed; .

DirectoryWatcher.prototype.doInitialScan = function doInitialScan() { fs.readdir(this.path, function(err, items) { if(err) { this.initialScan = false; return; } async.forEach(items, function(item, callback) { var itemPath = path.join(this.path, item); fs.stat(itemPath, function(err2, stat) { if(! this.initialScan) return; if(err2) { callback(); return; } if(stat.isFile()) { if(! this.files[itemPath]) this.setFileTime(itemPath, +stat.mtime, true); } else if(stat.isDirectory()) { if(! this.directories[itemPath]) this.setDirectory(itemPath, true, true); } callback(); }.bind(this)); }.bind(this), function() { this.initialScan = false; this.initialScanRemoved = null; }.bind(this)); }.bind(this)); };Copy the code

As we know from the above code, the contents of the current folder are read out when the first scan is performed. This.setfiletime (itemPath, +stat.mtime, true);

I won’t go into too much detail about setFileTime here, but it can be used in two different ways.

One from initialScan will read the latest modification time of all files to provide a basis for determining the file change trigger update. Another scenario is triggering an update.

Step2: directoryWatcher.watch((p, startTime))

DirectoryWatcher.prototype.watch = function watch(filePath, startTime) { this.watchers[withoutCase(filePath)] = this.watchers[withoutCase(filePath)] || []; this.refs++; var watcher = new Watcher(this, filePath, startTime); watcher.on("closed", function() { var idx = this.watchers[withoutCase(filePath)].indexOf(watcher); this.watchers[withoutCase(filePath)].splice(idx, 1); if(this.watchers[withoutCase(filePath)].length === 0) { delete this.watchers[withoutCase(filePath)]; if(this.path === filePath) this.setNestedWatching(false); } if(--this.refs <= 0) this.close(); }.bind(this)); this.watchers[withoutCase(filePath)].push(watcher); var data; if(filePath === this.path) { this.setNestedWatching(true); data = false; Object.keys(this.files).forEach(function(file) { var d = this.files[file]; if(! data) data = d; else data = [Math.max(data[0], d[0]), Math.max(data[1], d[1])]; }, this); } else { data = this.files[filePath]; } process.nextTick(function() { if(data) { var ts = data[0] === data[1] ? data[0] + FS_ACCURACY : data[0]; if(ts >= startTime) watcher.emit("change", data[1]); } else if(this.initialScan && this.initialScanRemoved.indexOf(filePath) >= 0) { watcher.emit("remove"); } }.bind(this)); return watcher; };Copy the code

This code records the creation of a Watcher by a Filepath and returns the wathcer.

So let’s go back

Watchpack.prototype._fileWatcher = function _fileWatcher(file, watcher) {
    watcher.on("change", function(mtime, type) {
        this._onChange(file, mtime, file, type);
    }.bind(this));
    watcher.on("remove", function(type) {
        this._onRemove(file, file, type);
    }.bind(this));
    return watcher;
};Copy the code

We can see that each file is bound to a change and remove event.

After a file is changed, directoryWatcher will initially listen to it and trigger the corresponding fileWatcher change event.

And _onChange is going to be called

Watchpack.prototype._onChange = function _onChange(item, mtime, file) {
    file = file || item;
    this.mtimes[file] = mtime;
    if(this.paused) return;
    this.emit("change", file, mtime);
    if(this.aggregateTimeout)
        clearTimeout(this.aggregateTimeout);
    if(this.aggregatedChanges.indexOf(item) < 0)
        this.aggregatedChanges.push(item);
    this.aggregateTimeout = setTimeout(this._onTimeout, this.options.aggregateTimeout);
};Copy the code

This triggers the Change event of the Watchpack instance, which is bound in the NodeWatchFileSystem.

// segment if(callbackUndelayed) this.watcher. Once ("change", callbackUndelayed); this.watcher.once("aggregated", (changes, removals) => { changes = changes.concat(removals); if(this.inputFileSystem && this.inputFileSystem.purge) { this.inputFileSystem.purge(changes); } const times = this.watcher.getTimes(); callback(null, changes.filter(file => files.indexOf(file) >= 0).sort(), changes.filter(file => dirs.indexOf(file) >= 0).sort(), changes.filter(file => missing.indexOf(file) >= 0).sort(), times, times); });Copy the code

So how do you trigger recompilation? The answers can be aggregated.

function example(err, filesModified, contextModified, missingModified, fileTimestamps, contextTimestamps) => {
        this.pausedWatcher = this.watcher;
        this.watcher = null;
        if(err) return this.handler(err);

        this.compiler.fileTimestamps = fileTimestamps;
        this.compiler.contextTimestamps = contextTimestamps;
        this.invalidate();
}Copy the code

The invalidate event is emitted because the _go event is executed again.

invalidate(callback) { if(callback) { this.callbacks.push(callback); } if(this.watcher) { this.pausedWatcher = this.watcher; this.watcher.pause(); this.watcher = null; } if(this.running) { this.invalid = true; return false; } else { this._go(); }}Copy the code