Front-end monitoring | how to implement dynamic acquisition and analysis of user behavior

This is the 50th original article without water. For more original articles, please visit 👆 and follow us. This article was first published on front-end Monitoring best Practices – Dynamic collection and analysis of user behavior

The opening is introduced

Hello, everyone, I am Qingyin, from the political cloud front-end team. Since last year, I have been responsible for the construction of user behavior collection and analysis system. I am very glad to have the opportunity to share with you our experience in building user acquisition and analysis system from 0 to 1.

The construction of value

First of all, why do we want to make such a user behavior analysis system?

Business value of data burial points

Students in the full department want more accurate positioning, and users frequently ask questions, reduce the amount of consultation, and improve user satisfaction. A series of functional modules have been launched to help users answer questions, but the effect is impossible to measure.

Operation wants to know which advertising space, which resource space is more valuable? Who are your target customers?

Product and design students want to improve the user experience. Development students in the continuous iteration of the functional version online. So what are the effects of what we thought were optimization points? How do you measure it? All need data indicators to support the next work.

We will be the named user behavior acquisition and analysis system for the armillary sphere, data collection service launched a year and a half, the number of logs is armillary sphere platform has reached 1.6 billion, the number of every working day to collect around 10 million, front internal dotted line is established interest groups, from gathering requirements, design plan to fall to the ground, The total manpower cost is around 90 person days.

Later, I will introduce the Armillary sphere system in four parts. The first part is the general introduction of the key architecture, and then it is divided into three modules of data acquisition, analysis and application.

The key structure

The overall flow of armillary sphere system can be summarized into three steps. First, the data is collected, then the data is processed, and finally, the data is presented.

To support these three departments, four functional modules are realized:

SDK for data acquisition
Data processing and data storage services
Pit-level data display Chrome plugin
A site for displaying system-level data

Overall architecture of the armillary sphere

Overall Architecture Preview

First, let’s take a look at the overall architecture diagram of the Armillary Sphere. We collected data from three terminals, PC, H5 and APP.

Two SDKS are used on PC and H5 to listen for different events. Then the monitored events are reported to the data processing service through the REST interface and stored in the Ali Cloud log service. We carried out test environment and true line environment, data isolation of the two environments. The SDK of the test environment will refresh in the form of negotiated cache, and the true line will use strong cache, and carry out version control; Due to the large amount of true line data, we regularly store the log database’s historical data of more than one and a half years to OSS. There is a scheduled task every day to filter a portion of the data to be stored in the database. It also interacts with many external systems.

Authority system: mainly control the access authority of armillary sphere site;
Performance system: output some high-frequency access pages for performance detection;
Operation and maintenance platform for system deployment; There will be some data interaction with the big data platform. We will push the behavioral data to the big data platform, and also extract some business-related data from the platform.

System Critical Architecture

See the detailed architecture diagram of the key modules above. The module on the left is for user behavior collection, while the module on the right is for internal users. Provide users with very rich data visualization. In addition to the visual site, it also provides Chrome plug-ins for data display. It also acts as a pass platform to provide some SQL queries, Excel reports export, and API data pulling reports. It can also carry out secondary development based on existing data.

The data collection

First of all, let’s talk about the realization of data acquisition module. We collected page entry and exit events, user click and scroll events, and some standard custom events. We can automatically collect the events of page entry, departure and scrolling. Click and custom events need to be implemented with the cooperation of front-end students and code implantation.

So how do we achieve automatic collection of zero code?

First of all, we implanted the project ID in the page through the project compilation process, and mounted the head part of the page
Then, when entering the page, according to the path of the page, to automatically generate the page ID, mounted on the body node
Finally, when the user enters and leaves the page, the SDK of data collection can automatically get the project ID and page ID to locate a unique page and automatically report the entry and departure events

Take the click event as an example. The DOM point that currently triggers the click event is called a pit. The DOM nodes that are wrapped around the pit are called blocks. Both need to be mounted in an intrusive manner, and to reduce the cost of mounting, we have provided tools to help them do so.

In the unified site, we provide the function of burying point application. After the application is completed, click generate code, but the location and ID to be mounted will be listed in detail immediately. Developers only need to paste the generated code to the location to be buried.

In addition, we talked about building in the last session. The page generated by the building system will also automatically implant these location ids. With these data, we can start a complete report.

Analysis of collected data

Here are some of the figures we presented at the conference. The xy coordinates here can be used to generate a thermal map of the user’s click area, reflecting the user’s main concerns on a page. The diagram below:

The operating system, resolution, and browser information allows us to analyze some of the major browsers of the current platform’s mainstream users to determine platform compatibility
If it is the APP side, there will also be some other data such as the client version, the phone model, the current network situation
LogType Classifies events
Evt to specify the event type
CreateTime is the time at which the event occurred and can be used to record the sequence of events
UtmCnt is the trigger location used to locate the current event to a specific DOM node
The UUID is the unique identifier of the visitor, and every time the user enters the page, the SDK will look for it in the cookie. If it doesn’t exist, we will generate it and set the expiration time to permanent. The IP address of the user can be traced back to the province or city of the user
A userId can be associated with a UUID to associate meaningless visitors with users of the platform to form a detailed user portrait
Utmfrom marks the location of the source, more on the jump to a link later, this field is used to concatenate the links before and after
The reported information in the green box can be summarized as the information related to the browser page, the information in the red box is the information related to the event, and the information in the blue box is the information related to the user

Event-specific interception

We delegate all four target events to the Document. All of these events will be blocked as soon as they are triggered, but after certain filtering, only events triggered by targets that can obtain pit IDS and block ids will be reported. This makes the data collected more accurate. If we do not conduct screening here, the reported data will become large and complete, and the amount of data will be extremely large. Therefore, the current code intrusive buried point will be switched to full buried point, which will also make the data analysis more difficult.

As you can see from the image above, a push request is sent by default when you first enter the page. There is also a push request for each click, but it is categorized in other.

This is because in the process of leaving, sending HTTP request is usually cancelled, and sendBeacon is a way to ensure that the data is sent. It can initiate an HTTP request in the unload or beforeunload event handler to send data, ensuring that our request to leave the page is recorded.

We also use ** tags in data reporting to ensure browser compatibility. At present, IE does not support sendBeacon method, and some users on our platform are still using IE, and we are constantly paying attention to the proportion of IE use, so the data of this part will be very important. In order to collect more IE usage data, we will first determine whether the sendBeacon method is available. When it is not available, ** will be used to send the request. Cors is the most common way for us to send cross-domain requests. It can use POST requests to successfully send some data reported in batches within the length limit.

Besides being able to send some reported data in a single dimension, we also need to be able to connect the whole browsing process of the user with the reported data. One of the most common is the jump of links.

Why not refer to connect the whole process? Mostly because his dimensions are too thick. He can only go to the source page and he can’t record where he came from on the page.

When we enter the page, we generate a unique ID for the next entry
When the user clicks a link, we will subscript the current project LED page ID block ID and the pit of the current LINK A
Finally, the ID generated when the page is generated is composed of five tag ids into a uTM value. After the page jumps, the SDK will obtain the uTM value from the URL of the page and report the uTM value

So that we can confirm that the source of the current page is from a page on which the location of the link, also let the link click behavior into an automated acquisition event, as long as the current was implanted the block ID, all of the below it a link click can be recorded, and the whole process used in tandem.

The data analysis

In the data processing part, we use the LOG Service of Ali Cloud. One of its great benefits is that it can provide the real-time consumption interface of logs, and the query means are very rich. Ability to add real-time indexes; It currently meets most of our query requirements.

In the process of data analysis, a very important point is how to convert the collected data into understandable indicators.

One of the main processes in the figure above is to conduct data analysis. Indicators should be set up first, then data analysis should be carried out after data of points to be analyzed can be obtained, and finally data is applied to the actual scene.

It’s easy to calculate basic metrics like PV, UV, CTR, and exposure based on the data we’ve collected, but how do we translate that into a funnel model?

And that’s to splice the underlying data. Take the flow loan process of our platform as an example:

After entering the application details page, we can use the UV that enters the page as the first data
Click immediately apply to open the protocol popup, the number of clicks as the second data, and so on, to assemble into a funnel model
We can calculate the percentage of the amount of each step relative to the previous step, which is an unordered funnel
Assuming that some users may enter directly from step 3, there may be situations where the percentage exceeds 100%
If it’s an ordered funnel, it filters on a user basis, so we’re going to filter from front to back and the users at each step are going to say, well, we’re only going to keep the users from the first step to the last step, not counting the users from the middle. This makes the conversion percentage must be less than or equal to 100%, and also makes the data more valuable for reference

Here is a scenario for setting confidence intervals:

Page stay time is actually the time difference between page entry and page departure. The fastest case is that the user leaves the page in 1s
However, it is also possible that the user interrupts the operation in the middle but does not close the window, resulting in a very long time
In the case of small sample size, such special data may push up the average page access time to more than 4 hours, and the average access time may lose reference value
The median here refers to 22 minutes and 8 seconds after sorting the duration of user visits from small to large. At this time, the median can better reflect the actual usage of the page

So how do we make the average time worth something?

We can set confidence intervals to exclude users who spend more than 5 or 8 hours on the page, depending on the situation, and then look at the average time spent on the entire page, or with the help of a bar chart, to see the overall distribution and trend of time spent.

Data applications

The current analysis model is mainly divided into four parts: event analysis, page analysis, transformation analysis and user analysis, and also divided into 10 kinds of small indicators. Tools are also provided in custom Settings to help establish metrics;

Event analysis is the basis of user behavior analysis and the most commonly used function, including frequency, distribution, and interval. You can create various analysis reports by analyzing events. Page based on the behavior data of each page, the page layout of the landing page is optimized to increase the attraction of the landing page. Thermogram analysis, by visualizing user behavior, helps us to deeply analyze user attention to content and features
Transformation analysis is the most important analysis model in user behavior analysis. Through transformation analysis, we can find out the transformation path and funnel of user behavior and improve the overall conversion rate of the platform. Conversion occurs all the way from the moment the user is reached to the moment the user completes the conversion
User analysis can well help us determine the target user group of the product, user behavior habits, user activity and retention characteristics, through user segmentation can achieve fine user operation

From the perspective of a single page, we can get many dimensions of data, which can be displayed on the page through plug-ins;

This is a data kanban, real-time view of customized core indicators and reports, real-time grasp of key indicators, to help users find problems in real time.

Operating system, browser ratio, you can click to check:

Number of visits, number of visits
Heat map: heat map is obtained by coordinate and resolution conversion
Path analysis: You can see where you’re coming from and where you’re going
Here is the capability for custom Settings: to create events and funnels

other

The Armillary sphere is a fairly basic data acquisition and analysis product, with a lot of extensible points behind it.

If the company does not need to build such a system by itself at the current development stage, but the business needs such a capability, it can also consider some third-party products. For the sake of platform data security, some of them also support localized deployment.

, recruiting

ZooTeam, a young passionate and creative front-end team, belongs to the PRODUCT R&D department of ZooTeam, based in picturesque Hangzhou. The team now has more than 50 front-end partners, with an average age of 27, and nearly 30% of them are full-stack engineers, no problem in the youth storm group. The members consist of “old” soldiers from Alibaba and NetEase, as well as fresh graduates from Zhejiang University, University of Science and Technology of China, Hangzhou Electric And other universities. In addition to daily business docking, the team also carried out technical exploration and practice in material system, engineering platform, building platform, performance experience, cloud application, data analysis and visualization, promoted and implemented a series of internal technical products, and continued to explore the new boundary of front-end technology system.

If you want to change what’s been bothering you, you want to start bothering you. If you want to change, you’ve been told you need more ideas, but you don’t have a solution. If you want change, you have the power to make it happen, but you don’t need it. If you want to change what you want to accomplish, you need a team to support you, but you don’t have the position to lead people. If you want to change the pace, it will be “5 years and 3 years of experience”; If you want to change the original savvy is good, but there is always a layer of fuzzy window… If you believe in the power of believing, believing that ordinary people can achieve extraordinary things, believing that you can meet a better version of yourself. If you want to be a part of the process of growing a front end team with deep business understanding, sound technology systems, technology value creation, and impact spillover as your business takes off, I think we should talk. Any time, waiting for you to write something and send it to [email protected]

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Front-end monitoring | how to implement dynamic acquisition and analysis of user behavior

The opening is introduced