Set up front-end anomaly monitoring system

Involves skills

  • Collect front-end errors (native, React, Vue)
  • Write error reporting logic
  • Write an error log collection service using egg.js
  • Write a Webpack plug-in to automatically upload Sourcemap
  • Restore the compressed source code location using Sourcemap
  • Use Jest for unit testing

The working process

  1. Collect wrong
  2. Report the error
  3. The code goes live and the Sourcemap file is uploaded to the error monitoring server
  4. The monitoring server receives and logs errors when they occur
  5. Error analysis is performed based on sourcemAP and error log content

Abnormal collection

Let’s start by looking at how to catch exceptions.

JS abnormal

Js exceptions do not cause the JS engine to crash, but only terminate the currently executing task. For example, a page has two buttons. If an abnormal page occurs when you click the button, the page will not crash, but the function of this button will be invalid, and other buttons will be effective.

setTimeout(() = > {
  console.log('1->begin')
  error
  console.log('1->end')})setTimeout(() = > {
  console.log('2->begin')
  console.log('2->end')})Copy the code

In the example above we started two tasks with setTimeout, even though the first task executed the wrong method. The execution of the program stopped. But the other task was not affected.

In fact, if you don’t open the console you won’t even see the error. It’s as if the mistake happened in silence.

Let’s look at how such errors can be collected.

try-catch

Our first thought was to use try-catch to collect.

setTimeout(() = > {
  try {
    console.log('1->begin')
    error
    console.log('1->end')}catch (e) {
    console.log('catch',e)
  }
})
Copy the code

If the error is not caught in a function, the error is thrown.

function fun1() {
  console.log('1->begin')
  error
  console.log('1->end')}setTimeout(() = > {
  try {
    fun1()
  } catch (e) {
    console.log('catch',e)
  }
})
Copy the code

The console prints error messages and error stacks, respectively.

Now, you might be thinking, well, why don’t you just make a try-catch error at the bottom. But the ideal is full, the reality is very skinny. Let’s look at the next example.

function fun1() {
  console.log('1->begin')
  error
  console.log('1->end')}try {
  setTimeout(() = > {
    fun1()

  })
} catch (e) {
  console.log('catch', e)
}
Copy the code

Notice that the exception is not caught.

And that’s because JS has very limited try-catch capabilities and it doesn’t work very well with asynchrony. You can’t add a try-catch to all asynchrony to collect errors.

window.onerror

The greatest benefit of window.onerror is that it can catch both synchronous and asynchronous tasks.

function fun1() {
  console.log('1->begin')
  error
  console.log('1->end')}window.onerror = (. args) = > {
  console.log('onerror:',args)
}

setTimeout(() = > {
  fun1()
})
Copy the code

  • Onerror return value

    One other problem with onError is that if you return true you don’t get thrown up. Otherwise you’ll see the error log in the console.

Listening for Error Events

Window. The addEventListener (‘ error ‘() = > {})

Well, onError is great but there’s still a class of exceptions that you can’t catch. This is a network exception error. Take the following example.

<img src="./xxxxx.png">
Copy the code

Imagine if the image we wanted to display on the page suddenly stopped showing and we didn’t even know it was a problem.

AddEventListener is

window.addEventListener('error'.args= > {
    console.log(
      'error event:', args
    );
    return true;
  }, 
  true // Use the capture method
);
Copy the code

Promise exception catching

Promise came along primarily to allow us to address the issue of callback geography. It’s basically standard for our program development. Although we advocate es7 async/await syntax for writing, we do not rule out that many ancestral codes still have Promise writing.

new Promise((resolve, reject) = > {
  abcxxx()
});
Copy the code

Neither onError nor listening error events can be caught in this case

new Promise((resolve, reject) = > {
  error()
})
// Add exception catching
  .catch((err) = > {
  console.log('promise catch:',err)
});
Copy the code

Unless each Promise adds a catch method. But obviously you can’t do that.

window.addEventListener("unhandledrejection".e= > {
  console.log('unhandledrejection',e)
});
Copy the code

We can consider unhandledrejection event capture error throw and let the error event handle

window.addEventListener("unhandledrejection".e= > {
  throw e.reason
});
Copy the code

Async /await exception capture

const asyncFunc = () = > new Promise(resolve= > {
  error
})
setTimeout(async() = > {try {
    await asyncFun()
  } catch (e) {
    console.log('catch:',e)
  }
})
Copy the code

In fact the async/await syntax is essentially a Promise syntax. The difference is that async methods can be caught by a superimposed try/catch.

If not, it will be captured with an unhandledrejection event, just like a Promise. In this case, we just need to add unHandlerejection globally.

summary

Exception types Synchronized methods Asynchronous methods Resource to load Promise async/await
try/catch ✔ ️ ✔ ️
onerror ✔ ️ ✔ ️
Error event listening ✔ ️ ✔ ️ ✔ ️
Unhandledrejection event listener ✔ ️ ✔ ️

In fact, we can throw the exception thrown by the unhandledrejection event again and we can handle it with the error event.

The final code is as follows:

window.addEventListener("unhandledrejection".e= > {
  throw e.reason
});
window.addEventListener('error'.args= > {
  console.log(
    'error event:', args
  );
  return true;
}, true);
Copy the code

Webpack engineering

Now is the era of front-end engineering, engineering exported code is generally compressed and confused.

Such as:

setTimeout(() = > {
    xxx(1223)},1000)
Copy the code

The error code points to the compressed JS file.

If you want to associate errors with the original code, you need the sourcemap file to help.

What is sourceMap

Simply put, sourceMap is a file that stores location information.

And, to be more careful, what this file contains is the position of the code after the transformation, and the corresponding position before the transformation.

How to use sourceMap to restore the location of the exception code will be covered in the exception analysis section.

Vue

Create a project

Create a project directly using vue-CLI tools.

Create a project vue create vue-sample CD vue-sample NPM I// Start the application
npm run serve

Copy the code

We’re going to turn off ESLint temporarily for testing purposes and it’s recommended that you turn esLint on at all times

Configure it in vue.config.js

module.exports = {   
  // Disable the ESLint rule
  devServer: {
    overlay: {
      warnings: true.errors: true}},lintOnSave:false
}
Copy the code

We intentionally in SRC/components/HelloWorld. Vue

<script>
export default {
  name: "HelloWorld".props: {
    msg: String
  },
  mounted() {
    // make an error
    abc()
  }
};
</script>Javascript window.addeventListener ('error', args => {console.log('error', error)})Copy the code

At this point the error is printed in the console, but the error event is not listened for.

handleError

To uniformly report Vue exceptions, use the handleError handle provided by Vue. This method is called whenever a Vue exception occurs.

We are in the SRC/main. Js

Vue.config.errorHandler = function (err, vm, info) {
  console.log('errorHandle:', err)
}
Copy the code

React

npx create-react-app react-sample
cd react-sample
yarn start
Copy the code

We make an error using useEffect hooks

import React ,{useEffect} from 'react';
import logo from './logo.svg';
import './App.css';

function App() {
  useEffect(() = > {	
    // An exception occurred
    error()
  });	

  return (
    <div className="App">/ /... A little...</div>
  );
}

export default App;
Copy the code

Add error event listening logic to SRC /index.js

window.addEventListener('error'.args= > {
    console.log('error', error)
})
Copy the code

But from the run result, although the output error log is still captured by the service.

ErrorBoundary label

Error bounds can only catch errors of its children. Error boundaries cannot catch errors of their own. If an error boundary cannot render the error message, the error bubbles up to the nearest error boundary. This is also similar to how catch {} works in JavaScript.

Create the ErrorBoundary component

import React from 'react'; 
export default class ErrorBoundary extends React.Component {
    constructor(props) {
      super(props);
    }
  
    componentDidCatch(error, info) {
      // Print an error when an exception occurs
      console.log('componentDidCatch',error)
    }
  
    render() {
      return this.props.children; }}Copy the code

Wrap the App tag in SRC /index.js

import ErrorBoundary from './ErrorBoundary'

ReactDOM.render(
    <ErrorBoundary>
        <App />
    </ErrorBoundary>
    , document.getElementById('root'));
Copy the code

In the last article we focused on how to collect JS errors. In this article we will talk about how exceptions are reported and analyzed.

Exception reporting

Select a communication mode

Dynamically create an IMG tag

The purpose of reporting is to send the captured exception information to the back end. The most common method is to create a label dynamically. There is no need to load any communication libraries and the page does not need to be refreshed. Basically including Baidu statistics Google statistics are based on this principle to do the buried point.

new Image().src = 'http://localhost:7001/monitor/error'+ '? info=xxxxxx'
Copy the code

By dynamically creating an IMG, the browser sends a GET request to the server. You can report errors to the server by placing the error data you need to report in a QueryString string.

Ajax report

We can actually use Ajax to report errors, just like we would in a business application. I won’t repeat it here.

What data to report

Let’s first look at the error event parameters:

The attribute name meaning type
message The error message string
filename Abnormal resource URL string
lineno Abnormal line number int
colno Abnormal column number int
error Error object object
error.message The error message string
error.stack The error message string

One of the core should be the error stack, in fact, we locate the error is the most important error stack.

The error stack contains most debugging information. It includes the exception position (row number, column number), exception information

Reported data serialization

Since the communication can only be transmitted as strings, we need to serialize the object.

There are basically three steps:

  • The exception data is deconstructed from the attributes and stored in a JSON object

  • Convert a JSON object to a string

  • Convert the string to Base64

And of course you have to do the opposite on the back end and we’ll talk about that later.

window.addEventListener('error'.args= > {
  console.log(
    'error event:', args
  );
  uploadError(args)
  return true;
}, true);
function uploadError({ lineno, colno, error: { stack }, timeStamp, message, filename }) {
    / / filter
    const info = {
      lineno,
      colno,
      stack,
      timeStamp,
      message,
      filename
    }
    // const str = new Buffer(JSON.stringify(info)).toString("base64");
  	const str = window.btoa(JSON.stringify(info))
    const host = 'http://localhost:7001/monitor/error'
    new Image().src = `${host}? info=${str}`
}
Copy the code

Abnormal collection

The abnormal data must be received by a back-end service.

Take eggJS, a popular open source framework, as an example

Set up the EGGJS project

# create backend project egg-init backend --type=simple CD backend NPM I # Start project NPM run devCopy the code

Write the error upload interface

Start by adding a new route to app/router.js

module.exports = app= > {
  const { router, controller } = app;
  router.get('/', controller.home.index);
  // Create a new route
  router.get('/monitor/error', controller.monitor.index);
};
Copy the code

Create a new controller (app/ Controller /monitor)

'use strict';

const Controller = require('egg').Controller;
const { getOriginSource } = require('.. /utils/sourcemap')
const fs = require('fs')
const path = require('path')

class MonitorController extends Controller {
  async index() {
    const { ctx } = this;
    const { info } = ctx.query
    const json = JSON.parse(Buffer.from(info, 'base64').toString('utf-8'))
    console.log('fronterror:', json)
    ctx.body = ' '; }}module.exports = MonitorController;

Copy the code

Log file

The next step is to log errors. The implementation of the method can be written in FS, or with the help of log4JS such a mature log library.

Of course eggJS supports custom logging, so you can use this feature to customize a front-end error log.

In the/config/config. Default. Add a custom js logging configuration

// Define the front-end error log
config.customLogger = {
  frontendLogger : {
    file: path.join(appInfo.root, 'logs/frontend.log')}}Copy the code

In the/app/controller/monitor. Js add logging

async index() {
    const { ctx } = this;
    const { info } = ctx.query
    const json = JSON.parse(Buffer.from(info, 'base64').toString('utf-8'))
    console.log('fronterror:', json)
    // Log the error
    this.ctx.getLogger('frontendLogger').error(json)
    ctx.body = ' ';
  }
Copy the code

Abnormal analysis

When it comes to exception analysis, the most important work is actually to restore the webPack obfuscated compression code.

The Webpack plug-in implements SourceMap uploads

The Sourcemap file is generated during webpack packaging and needs to be uploaded to the exception monitoring server. This function we use the Webpack plug-in to complete.

Create the WebPack plug-in

/source-map/plugin

const fs = require('fs')
var http = require('http');

class UploadSourceMapWebpackPlugin {
  constructor(options) {
    this.options = options
  }

  apply(compiler) {
    // Execute after packing
    compiler.hooks.done.tap("upload-sourcemap-plugin".status= > {
      console.log('webpack runing')}); }}module.exports = UploadSourceMapWebpackPlugin;
Copy the code

Load the WebPack plug-in

webpack.config.js

// Automatically upload the Map
UploadSourceMapWebpackPlugin = require('./plugin/uploadSourceMapWebPackPlugin')

plugins: [
    // Add automatic upload plugin
    new UploadSourceMapWebpackPlugin({
      uploadUrl:'http://localhost:7001/monitor/sourcemap'.apiKey: 'xxx'})].Copy the code

Add read sourcemAP read logic

Add logic to read sourcemap files in apply

/plugin/uploadSourceMapWebPlugin.js

const glob = require('glob')
const path = require('path')
apply(compiler) {
  console.log('UploadSourceMapWebPackPlugin apply')
  // Definitions are executed after packaging
  compiler.hooks.done.tap('upload-sourecemap-plugin'.async status => {
    // Read the sourcemap file
    const list = glob.sync(path.join(status.compilation.outputOptions.path, `./**/*.{js.map,}`))
    for (let filename of list) {
      await this.upload(this.options.uploadUrl, filename)
    }
  })
}
Copy the code

Implement the HTTP upload function

upload(url, file) {
  return new Promise(resolve= > {
    console.log('uploadMap:', file)
    const req = http.request(
      `${url}? name=${path.basename(file)}`,
      {
        method: 'POST'.headers: {
          'Content-Type': 'application/octet-stream'.Connection: "keep-alive"."Transfer-Encoding": "chunked"
        }
      }
    )
    fs.createReadStream(file)
      .on("data".chunk= > {
      req.write(chunk);
    })
      .on("end".() = >{ req.end(); resolve() }); })}Copy the code

Add an upload interface on the server

/backend/app/router.js

module.exports = app= > {
  const { router, controller } = app;
  router.get('/', controller.home.index);
  router.get('/monitor/error', controller.monitor.index);
  // Add an upload route
 router.post('/monitor/sourcemap',controller.monitor.upload)
};
    
Copy the code

Add the SourcemAP upload interface

/backend/app/controller/monitor.js

async upload() {
    const { ctx } = this
    const stream = ctx.req
    const filename = ctx.query.name
    const dir = path.join(this.config.baseDir, 'uploads')
    // Determine whether the upload directory exists
    if(! fs.existsSync(dir)) { fs.mkdirSync(dir) }const target = path.join(dir, filename)
    const writeStream = fs.createWriteStream(target)
    stream.pipe(writeStream)
}
Copy the code

The call plug-in Sourcemap is uploaded to the server when performing webPack packaging.

Parsing ErrorStack

Considering that this feature requires a lot of logic, we are going to develop it as a stand-alone function and use Jest for unit testing

Let’s look at our requirements

The input A stack of errors ReferenceError: xxx is not defined\n’ + ‘ at http://localhost:7001/public/bundle.e7877aa7bc4f04f5c33b.js:1:1392’
SourceMap slightly
The output Source error stack { source: ‘webpack:///src/index.js’, line: 24, column: 4, name: ‘xxx’ }

Build the Jest framework

Start by creating a /utils/ stackParser.js file

module.exports = class StackPaser {
    constructor(sourceMapDir) {
        this.consumers = {}
        this.sourceMapDir = sourceMapDir
    }
}

Copy the code

Create the test file stackParser.spec.js in the sibling directory

The above requirements are expressed by Jest

const StackParser = require('.. /stackparser')
const { resolve } = require('path')
const error = {
    stack: 'ReferenceError: xxx is not defined\n' +
        ' at http://localhost:7001/public/bundle.e7877aa7bc4f04f5c33b.js:1:1392'.message: 'Uncaught ReferenceError: xxx is not defined'.filename: 'http://localhost:7001/public/bundle.e7877aa7bc4f04f5c33b.js'
}

it('stackparser on-the-fly'.async() = > {const stackParser = new StackParser(__dirname)

    / / assertions
    expect(originStack[0]).toMatchObject(
        {
            source: 'webpack:///src/index.js'.line: 24.column: 4.name: 'xxx'})})Copy the code

Now let’s run Jest

npx jest stackparser --watch
Copy the code

The display fails simply because we haven’t implemented it yet. So let’s implement this method.

Antisequence Error object

First create a new Error object to set the Error stack to Error, then use the error-stack-parser NPM library to convert to stackFrame

const ErrorStackParser = require('error-stack-parser')
/** * error stack deserialization *@param {*} Stack Error stack */
parseStackTrack(stack, message) {
  const error = new Error(message)
  error.stack = stack
  const stackFrame = ErrorStackParser.parse(error)
  return stackFrame
}
Copy the code

Parsing ErrorStack

Next we convert the code location in the error stack to the source location

const { SourceMapConsumer } = require("source-map");
async getOriginalErrorStack(stackFrame) {
        const origin = []
        for (let v of stackFrame) {
            origin.push(await this.getOriginPosition(v))
        }

        // Destroy all consumers
        Object.keys(this.consumers).forEach(key= > {
            console.log('key:',key)
            this.consumers[key].destroy()
        })
        return origin
    }

    async getOriginPosition(stackFrame) {
        let { columnNumber, lineNumber, fileName } = stackFrame
        fileName = path.basename(fileName)
        console.log('filebasename',fileName)
        // Check whether it exists
        let consumer = this.consumers[fileName]

        if (consumer === undefined) {
            / / read sourcemap
            const sourceMapPath = path.resolve(this.sourceMapDir, fileName + '.map')
            // Check whether the directory exists
            if(! fs.existsSync(sourceMapPath)){return stackFrame
            }
            const content = fs.readFileSync(sourceMapPath, 'utf8')
            consumer = await new SourceMapConsumer(content, null);
            this.consumers[fileName] = consumer
        }
        const parseData = consumer.originalPositionFor({ line:lineNumber, column:columnNumber })
        return parseData
    }
Copy the code

Let’s test that out with Jest

it('stackparser on-the-fly'.async() = > {const stackParser = new StackParser(__dirname)
    console.log('Stack:',error.stack)
    const stackFrame = stackParser.parseStackTrack(error.stack, error.message)
    stackFrame.map(v= > {
        console.log('stackFrame', v)
    })
    const originStack = await stackParser.getOriginalErrorStack(stackFrame)

    / / assertions
    expect(originStack[0]).toMatchObject(
        {
            source: 'webpack:///src/index.js'.line: 24.column: 4.name: 'xxx'})})Copy the code

Log the source location

async index() {
    console.log
    const { ctx } = this;
    const { info } = ctx.query
    const json = JSON.parse(Buffer.from(info, 'base64').toString('utf-8'))
    console.log('fronterror:', json)
    
    // Convert to source location
    const stackParser = new StackParser(path.join(this.config.baseDir, 'uploads'))
    const stackFrame = stackParser.parseStackTrack(json.stack, json.message)
    const originStack = await stackParser.getOriginalErrorStack(stackFrame)
    this.ctx.getLogger('frontendLogger').error(json,originStack)
    
    ctx.body = ' ';
  }
Copy the code

Open source framework

Fundebug

Fundebug focuses on real-time BUG monitoring for JavaScript, wechat applets, wechat games, Alipay applets, React Native, Node.js and Java online applications. Since its official launch on November 11, 2016, Fundebug has handled over 1 billion error events in total, and paid customers include Sunshine Insurance, Lychee FM, Zhangmen 1-on-1, Walnut Programming, Weomai and many other brand enterprises. Welcome free trial!

Sentry

Sentry is an open source real-time error tracking system that helps developers monitor and fix exceptions in real time. It focuses on continuous integration, improving efficiency, and improving the user experience. Sentry is divided into server and client SDK. The former can directly use the online services provided by its home, or can be built locally. The latter provides support for many major languages and frameworks, including React, Angular, Node, Django, RoR, PHP, Laravel, Android,.NET, JAVA, and more. It also offers solutions to integrate with other popular services, such as GitHub, GitLab, Bitbuck, Heroku, Slack, Trello, and more. The company’s current projects are also phasing in Sentry for error log management.

conclusion

So far, we’ve taken the basic functionality of front-end exception monitoring as an MVP(minimum viable Product). There is much more to be done, and ELK can be used for analysis and visualization of error logs. Docker can be used for publishing and deploying. Add permission control to upload and report EggJS.

Reference code location: github.com/su37josephx…