The fuse

One day, a test colleague’s mobile page went blank. It seemed that there was something wrong with the page. When I opened the page by myself, I did not report any error, but finally found that the error only existed in his mobile phone, and the mobile terminal project was in the wechat environment, so it would be troublesome to debug. Finally, I used his mobile phone to debug the problem: there was something wrong with the message data of a conversation under his account, which led to the error of the page. In general, the problem can be found quickly only by using his mobile phone or account for debugging. What if it is an external user? I can’t take his mobile phone to test.

 

Actually this question is very common, but this time I think of this problem if we didn’t colleagues found, then, may waste more energy to find out problems, even lead to a serious online bug, consider very fear, chengdu FCC just shortly before the big front leaf languages of monitoring the communication meeting, also let me inspiration, These public services are the core wealth of the company. At present, the business development of the company is in a rising stage, and there will definitely be more and more users in the future, with higher and higher requirements on the stability of the system. Since we still lack this service, it is appropriate to do it now.

 

preparation

From the beginning of putting forward this idea, I knew that landing is the key, otherwise all empty talk. Just half a month later, our front end team needed to do a sharing in the company, I now do a theme is quite suitable for sharing, other backend and testing colleagues also easy to listen to a little bit. Initially I looked at back-end storage and visualization, looking for an off-the-shelf backend integration tool to help me with the back-end. Elasticsearch+Fluentd+Kibana After a bit of research, there was always something wrong, but after some research, I realized that maybe I still need to do some custom development to solve the requirements, which is what my back-end colleagues said when they heard about my requirements. One person is limited, and the company has a lot of business matters, so it is very good to find a back-end colleague to cooperate, and make use of their advantages to achieve faster implementation, so THAT I can also focus on the front-end work and control the whole project implementation. So I talked it over with my backend colleague, and he agreed to work with me sometime. Leaving behind the back-end, I began to think about the front-end work, to investigate other people’s plans and knowledge of this area. There are some third-party libraries or open source projects that provide similar functionality and do a very simple understanding. Finally, I want to develop my own business which is easier to adapt to my own business, and the first version of the demand function is not so much development volume, then do it by myself. In the early stage, I met some functional points that need to be solved and implemented: generating sourcemap, monitoring JS errors and information reporting, sourcemap parsing after reporting compressed JS code, how to apply it more smoothly in business projects, data storage optimization, etc.

 

Basic implementation

The front end

  • Js error event monitoring + processing report
  • The build tool generates the Sourcemap file
  • Sourcemap file upload

The back-end

  • Provides an interface collection error
  • Read sourcemap file, parse upload error (parsing time: processing immediately after interface collection, processing later extraction)
  • Store the data

 

Monitor JS errors and information reporting

Through onError we can listen to and get js error messages, can get the following code five parameters. ColumnNo, error may not be available in some older ie8-9 browsers, opera and other browsers, but it does not matter if the columnNo, error parameters are not available. ColumnNo and error parameters of older browsers can also be obtained by other means. Currently, monitoring is mainly for mobile terminals, and there is no need to be compatible with older browsers.

window.onerror = function (msg, fileUrl, lineNo, columnNo, error) {}

 

The onerror method is roughly implemented as follows:

There may be cross-domain problems. Js in different domains need to configure script attribute Crossorigin =”anonymous” and back-end configuration access-Control-allow-Origin. However, there is no cross-domain problems in our project at present.

Now we can get the error message through onError, but the code on the line is compressed, and the number of lines and columns and variable lives we can get when reporting an error do not tell us where the source code went wrong. Here we need sourcemap, which we’ll talk about.

 

sourcemap

Sourcemap is an information file that stores location information. That is to say, sourcemap file records the position before the code conversion and transformation after the corresponding position (www.ruanyifeng.com/blog/2013/0)… . Figure 1 below is a compressed version of login.js. The comment in the second line specifies the relative path to the map file. Figure 2 is a map file (json format); Figure 3 Figure 4 shows the Sourcemap file. Figure 2. The generated map file sourcesContent field directly imports the source code (the build tool can configure whether to import source files into the map file or not), which makes it easy for the back end to parse, because without the source file, the back end will not get the correct result.

(figure 1)

(figure 2)

(figure 3)

 

 

(figure 4)

 

 

Grunt generates sourcemap:

Our mobile build tools are older and use Grunt as the packaging tool. SourceMap has not been used before for code compression, and because the development and test environments do not have compression, there is no need to use sourceMap for debugging in the browser. Then I went back to the gruntfile file (I didn’t write it before), sourceMap configuration didn’t seem to match the official documentation, kept getting errors, and finally realized that the previous version of the packaging tool was 13 years old, so there was no need to deal with the version problem for the time being. The sourcemap file is successfully generated in the same directory as the source file, for example, the source file is called xx.js and the map file is xx.js.map. We add md5 version numbers to the js files, so the actual files are xx.md5.js and xx.md5.js.map(MD5 varies by content).

 

Sourcemap parsing problem

The biggest difficulty when thinking about it should be the sourcemap parsing. Sourcemap is a data format that doesn’t matter in the development language. If you know the principle of Sourcemap, you should know that sourcemap is a data format that doesn’t matter in the development language. I handed the map file and error message to my colleagues at the back end, and they successfully parsed out the answer using a tool of go language, and realized the local file parsing. However, what we need is automatic parsing. It is impossible to manually take out the stored error information every time and then find the corresponding map file for manual parsing. Therefore, we need to find the map file by ourselves, and resolve the error message.

As a result, there are two key problems with back-end parsing:

  1. Where is the map file stored
  2. When to parse

1 Where is the map file stored

Here only say that our scheme, the map files and source js file package to the directory, at the same level with uploaded to the server (such as the path of the js is www.xxx.com/dist/index.md5.js, The map file’s address is www.xxx.com/dist/index.md5.js.map), the server can be according to the error of js path plus. The map suffixes to find the map file. SourceMappongURL specifies the location of the map file. After opening the browser, the debugger will find the map file, and the source code will be available in the browser. To avoid this, the server must make files with the.js.map suffix inaccessible. In this case, the server cannot directly download the static resource. Map file, but needs to find the corresponding map file on the server, which is troublesome to configure the path and write logic separately, and it is not flexible to change the folder structure. Therefore, our solution is to verify the token permission. The map file must be added with correct token parameters before the server returns resources (xxx.js.map? Token = XXXX), otherwise Nginx will block requests with no token or incorrect token.

 

② When to parse

There are two methods: One is that after receiving the error message, the back-end interface immediately finds the map file and parses it and stores it in the database. One way is to reserve the reported information and parse it through the interface. We choose the former. After the interface receives the data, the backend checks whether the current file has been downloaded locally according to the URL of the file reporting the error. If the file already exists, it will directly use the local file to resolve it. Then read the current local file and parse it. The parsed data and reported data are saved as a record. If it is the latter method, there are a lot of troublesome problems, not to mention here.

Here’s a diagram detailing our parsing flow:

 

One thing can happen: The current project has been updated to version 1.1, and an error of version 1.0 was not triggered before. At this time, a user caches the code of version 1.0 and triggers a new error. At this time, the map file stored locally by the server does not contain this file, so they will download the map file with token. Because the current version is 1.1, the original JS file has been changed, and the MD5 version does not correspond to the map file. In this case, the map file cannot be found and resolved. Therefore, in this special case, only the errorInfo information reported can be stored.

 

How to apply smoother in business projects

At present, the onerror method of JS has only a small amount of code, and there will be superposition in the later stage. Now the idea is to try not to do too much contact with the business code, just need to directly introduce the current JS to the various business projects, each project does not need to configure it too much, let it as simple as possible.

 

Storage optimization

In the later period, the management background will be used to query and statistics these abnormal logs. The same error may upload the error data to the server, and the back end will query one independent record. We cannot distinguish whether the error of this record contains repeated data, and we should not let the back end do field comparison. Later, I came up with the idea of combining the file path + line + column information for MD5 generation, and generated MD5 according to this unique value. Finally, when querying, I only need to query the current MD5 field to know how many records there are in this error. But I think too naive, different browser error information is a little different, the same error may generate different md5 string, even a little problem here, I still continue to use this solution to save the md5 (because the kernel, move the difference is small, the current field can also have some distinct).

The main data stored in our first release (and some general ones not to mention) :

{
    "businessInfo": "{}",//Custom data for business items
    "errorMd5": "80bb86b86da0607c0dc5c3a77e16eab6",//Md5 generated based on the error information
    "manualSendError": "{}",//Error information manually uploaded. Procedure
    "pageUrl": "http://www.xxx.com/xxx.html",//Release the page URL reporting an error
    "parseError": true.//Explain failure
    "parsed": '{"col":0,"errKey":"list","file":"xxx.js","line":105}',//Parsed columns, file paths, and variables
    "raw": '{"msg":'', "fileUrl":'', "lineNo":'', "columnNo":'', "error":''}',//Five arguments to onerror"UserAgent" : "Mozilla / 5.0 (Linux; Android 6.0.1; Redmi 4X Build/MMB29M; wv)..."//navigator.userAgent
}

 

 

Send E-mail

Email reminder is a very necessary function, at present has realized the real-time email reminder function. Set up a separate mailbox called frontendMonitor@. When the backend interface receives an error message, it sends the resolution data to the front-end through this mailbox to remind the front-end. If you use QQ email or personal email, you need to enable SMTP service in your account. By default, this function is enabled for QQ enterprise email. Pay attention to performance and optimization problems in the mail function, and do not cause the server to hang up because of too many front-end errors.

 

Optimization after actual use

  • In chrome and Firefox, the columnNo parameter is slightly different. Console 18 0 true; testBase 18 0 true; console 18 0 true; testBase 18 0 true; console 18 0 true The console. The log (testBase). TestBase = undefined, testBase = undefined, testBase = undefined, testBase = undefined Although don’t care about IE, but IE11 error column number and Firefox consistent.

　　　　

  • The page triggers an event, and the user keeps triggering the button. In this case, error messages will be continuously reported. Solution: Store the last error message and time, compare, the same error, in a short period of time to avoid repeated sending.
  • Window. onerror will not be triggered. You need to use the global listener of the framework itself to capture the information and manually upload it. Here you need to add the method of manually uploading the error message.
  • For monitored projects, some service information may need to be uploaded to facilitate analysis. Therefore, a configuration field is reserved for incorrect upload requests.

 

conclusion

These non-business services come from personal interest and thought, and there is no pressure from above to do them or when to do them. Start with an idea, research it, ask your backend colleagues for help, start and finish. I need to stick to this process, because I am afraid that I will not be able to land in the end, so I seize the time to realize every detail of the idea step by step, so that things can be landed and online as soon as possible, so as not to delay the matter longer. As the demand side, a better grasp of the whole project, combined with their own interests, so this is a bit to learn the language myself, ensure that can read the back-end code and understand the back-end logic, it is best to do a bit of development, on the basis of the code in the back-end colleagues, implements the email function, I call it a light into the shallow, finishing loading force ran road ~ now has launched the first edition, Just online and in less than two hours, he received an error mail, frighten I quickly find a bug, soon found out the problem, this bug should exist for a long time, but because there is no obstructive, and there is no impact to the business, also hasn’t been found, the conclusion is that we the front-end abnormal monitoring function is successful! There are still many functions to be developed in the later stage, such as statistics, data visualization, intelligent alarm and so on. The first edition landed, laying a good foundation for future iterations and evolution.

In the process of doing this, I wanted to put the project into practice as soon as possible, and time was limited, and I did not do enough research, such as how some existing open source projects are done. Later learned from colleagues sentry these three sides open source projects, also a little bit lost, although I also solved my needs, but the three open source project is a very perfect system, provides many functions, the much more powerful than me, then I do this what meaning, feel completely can’t compete with others, Will I continue to iterate on this project in the future? Is it necessary to continue to iterate? In the future, when there are special customization requirements, maybe it is easier to adapt to the business by developing your own, but is there a chance? This landing has met my initial requirements, and it can also help me solve the current problems. There are still many challenges and iterations waiting for me in the future. Will I take it all the way or die on the way? I want to say:

 

Finally, I would like to thank our backend colleagues for their support!! ~

 

Follow the official account of the great poet, the first time to get the latest articles.