Engaged in the Web front-end students must have deep experience in this, after the code is issued, like a runaway horse, running in thousands of customer terminals, until the product and background feedback problems to our side, many times the positioning problem can only rely on guessing, especially some accidental incentives, because do not know how the user is operating, Problems encountered in the real environment are usually formed by the superposition of many random factors, so it is difficult to replay the user’s operation to restore the site to find the cause.

The pain point right now

  • After the user reported the problem, although we queried the JS error log and cgi calls, we did not find any error, we have no idea what happened on the user’s terminal.
  • The JS error reporting system currently used has a certain delay. As a financial product, every minute counts. This short delay may be fatal, so we must quickly find the cause and repair it.

Over time, we realized that it would be a great help if we had a code-run log as detailed and searchable as the background.

Our quest

A robust logging system generally consists of three main parts: logging, uploading, and analyzing logs, and in this practice we have explored all three.

Where are the logs?

Because of the limitations of the front end, the file can not be accessed in the file sandbox as the APP can, so how to store the log persistently becomes a problem. Currently H5 also supports several localStorage solutions, such as cookies, localStorage, indexedDB and websql, etc. Other capabilities provided by plug-ins are not considered, such as flash, mobile usability will be a big issue.

What is an elegant log format?

  • Timestamps, needless to say, are fundamental to logging.
  • Multi-session: There may be multiple independent modules in a front-end project. These modules may record their own logs without interfering with each other. If each module logs under its own session, it does not interfere with each other and provides another dimension of filtering capability.
  • Error level, such asinfo.warning.error.criticalEtc., to provide the most basic filtering capacity.
  • The descriptor, such asverify.request.start.submit.preventedAnd so on. As much as possible, the user can know the general content of the log at a glance, which also facilitates the readability of the logging code in the code, and makes it easier to search in the code.
  • Data and details, sometimes a simple descriptor does not explain the problem, and more descriptive text or data may need to be recorded to analyze the problem.

How do I get logs?

Since we do not need to obtain a large number of logs from the client in real time, so the logs are stored locally in the user client, we need to cooperate with the background to develop an interface for receiving and storing the log content uploaded by the client through the network. There may also be security issues to consider, such as introducing token mechanisms and verifying login states. At present, the scheme adopted by Tencent Micro Securities is as follows: users send message “problem feedback” in wechat public account (direct use of key words such as log reporting may cause users’ aversion), and the background will return the address link of the log uploading page. After users click the link to enter, they will automatically report the log after obtaining the login status. We can follow Tencent micro securities public account on wechat to try.

How to analyze logs?

Since Logline uploads logs in standard and readable formats, they can be read to some extent directly using command-line tools or editors. However, users unfamiliar with the command line may still struggle, so it is necessary to use the Web technology stack to build an easy-to-use and visually pleasing tool. We hope that this tool can not rely on the backend, can be deployed on the server side, can be opened as a local web page directly by double clicking, or can be simply wrapped as a shell and used as a desktop APP.

As a log, the main content is a lot of plain text. After investigating some solutions, we think that FileReader. ReadAsText in the H5 specification can do this very well. Users can drag and drop one or more log files to a web page to analyze and retrieve these logs in batches.

We built an experience-only version of logline-Viewer.

Logline

Based on the above work, our Tencent Micro Securities project team launched its own solution: Logline, a lightweight, practical and client-level front-end logging tool.

Application scenarios

  • Core Process monitoring

    In some core processes of the product, we can actively upload user logs in the case of user errors, so that we can quickly count and locate the problems encountered by users.

  • Proactively capture user logs to analyze user behaviors

    Sometimes when users don’t cooperate with the developer, we can design a strategy that we released a json file online, for example, in a hope to take the initiative to grab log list of users, when our products on the user’s phone is open, the time delay download the json (avoid affect the main process performance), when matching the current user, Reports logs of the user.

  • Statistical and auxiliary analysis of JS errors

    We can log js errors, including the call queue, upload the error log directly or upload it uniformly when the accumulated threshold is reached.

Feature supports

  • Core logging capabilities
  • The namespace
  • Websql localStorage/with indexedDB three storage solution
  • classification
  • Log cleaning
  • Upload the log
  • No external dependencies

Quick learning

1. Import the script and select the protocol to be used

There are currently three protocols supported, all of which are mounted directly on the Logline object for special application scenarios and better semantics:

  • websql: Logline.PROTOCOL.WEBSQL
  • indexeddb: Logline.PROTOCOL.INDEXEDDB
  • localstorage: Logline.PROTOCOL.LOCALSTORAGE
Var Logline = require('./mod/logline.min'); Logline.using(Logline.PROTOCOL.WEBSQL); // Script TagCopy the code

2. Clear logs

Logline.keep(.5); // Retain logs generated within half a day. If no parameter is passed, clear logline.clean (); // Clear the log and delete the databaseCopy the code

3. Record logs

Var spaLog = new Logline('spa'), sdkLog = new Logline(' SDK '); SpaLog. Info ('init.succeed'); SpaLog. Error ('init.failed', {retcode: 'EINIT', retmsg: 'invalid signature'}); // Sdklog. warning('outdated'); Sdklog. critical('system.vanish', {// debug infos here});Copy the code

4. Read logs

Logline.getAll(function(logs) {
    // process logs here
});Copy the code

Custom build

NPM Run configure implements the localStorage, WebSQL, and IndexedDB logging protocols. By default, Logline is packaged as localStorage, WebSQL, and IndexedDB. This helps to reduce the package size.

NPM run configure // Configure the protocol you need, --with-xxx NPM run configure -- --with-localstorage --with-websql --with-indexeddb // Go to the dist directory to find the newly built package fileCopy the code

Cases used in Tencent Micro securities

Account opening Process Monitoring

Account opening process, as one of the core process, has always been a more perfect monitoring data, after once revised, we found that the user accounts submit failure ratio has obvious rise, many users submit your informations in the final, the back-end check found some user data has not been set up this kind of situation very confuse us. That’s what prompted us to develop this tool. After deploying the Logline solution, we added more detailed logging to the code of the registration process and proactively reported logs to the back end in case of errors. In some models, after a user clicks Next, the loading mask is slowly displayed globally. As a result, the user clicks next twice in a row. As a result, the controller of the registration process triggers the next step twice in a row, causing some pages to be skipped.

Fix JS error

At present, like many front-end products of Tenpay, we have access to tenpay unified log reporting platform, but the content we upload is very limited, the access performance is not very good, and we can’t see the WRONG JS call queue, or even the user’s Useragent. After using our own Logline reporting system, all these problems were solved. We found some errors generated by wechat JSSDK itself, such as “WeixinJSBridge is not defined” and some JS errors with a large number of reports, which effectively guaranteed the quality of the code. You can even do an error real-time curve statistic.

Configuration-based active fetching

As the product is in rapid functional iteration, both the business code and the code of the core module are often changed, which inevitably leads to some unexpected problems. Sometimes, we go to users and ask them to help us locate problems, but not all users are willing to cooperate. Therefore, we designed a set of strategies for actively capturing user logs: after our product page is opened, a logcat.json file will be downloaded after a certain time delay, and the file will contain a list of users we expect to actively report logs. When it matches the user, the user’s logs will be uploaded automatically. We only need to maintain such a configuration file when needed, and we will receive an email notification when the user logs in the configuration are uploaded. You can also use this configuration file to configure the client JS to automatically report JS errors.

CONTRIBUTING

We believe the front-end community should be open and collaborative.

Logline – Log recording and reporting module

logline

Logline-viewer – Log analysis platform

logline-viewer

WIP

[√] Added support for the IndexedDB protocol [√] Stripped log upload implementation to expose the method of getting all logs instead of the user’s own decision on the upload path [√] Allows custom compilation, packages only the protocol you need to reduce the size of the packed file [] Does not use Webpack compilation, To reduce the size of the packaged file

The future planning

In this practice, we deliberately kept the module independent and did not use any external dependencies, so it was easy to reuse to other businesses or bind our own business logic. In the future, however, we prefer Logline to be offered as a complete solution for front-end logging, without the business having to do its own implementation in the upload and analysis parts.

A more useful application scenario would be for the business to apply for an AppID on our platform and then use our statistical script, so the logs will be uniformly reported to our platform and the business side can use this APPID to browse, analyze and retrieve the required logs in our system.

The technique and implementation difficulty of this scheme are not difficult, but the important thing is a way to solve the pain point.