The SDK is an open source project, and any person or organization can see our source code on GitHub (github.com/sensorsdata…

Our SDK covers Android, iOS, Web, and mainstream back-end development languages such as Java, C++, C, PHP,.NET, Node, Go, Python, Ruby, etc., as well as small programs such as wechat, Baidu, Bytedance, Alipay, QQ, kuaiapp, etc. It also covers mainstream cross-platform development frameworks such as React Native, APICloud, mPaaS, Flutter, etc.

Common application scenarios of data analysis:

• User portrait
• Personalized recommendation
• The cheating
• Accurate advertising
• Online analytical
• Search optimization
• Text mining

The data types of data collection can be divided into two dimensions.

By data type: user behavior data, user-related data, service-related data, and text data

According to the owner of data: first party data, third party data

Is data collection important? Since the establishment of Shence 5 years ago, we have served more than 1000 enterprise customers, we found a phenomenon: many customers product iteration speed is very fast, basically a version every two weeks, or even a version every week. In such a short period of time, it is not easy to do business development and to bury it. If the two conflict, more customers will choose to give up burying point.

Data analysis process is probably: data acquisition and data transfer (real time/batch) to data modeling (storage), data statistics/analysis/mining and data visualization (feedback), from this process we can find that data acquisition is the basic data analysis, is the source, data acquisition and business development are equally important.

What would a complete burying process look like? The overall process is similar to product development.

In Shence analysis, we use Event model to describe various behaviors of users on the product, which is also the core basis of all interfaces and function design of Shence analysis. Simply put, the Event model consists of two core entities, Event and User.

An Event is who did what, when, where, and how. For the User, User is the subject of this event.

When carrying out the burying point, we should follow the four principles of “big”, “full”, “fine” and “time”.

“Large” mainly takes into account that with the development of our business, the scale of users and data will grow exponentially. It is not only large in volume, but also an important prerequisite for us to accumulate data assets.

“Complete”, mainly refers to multi-terminal collection. For example, There are many kinds of SDK of Shence, which will cover mainstream platforms and development languages. Only in this way, when we do data analysis, we can do full user behavior analysis, rather than sampling analysis. At the same time, we can also collect user behavior data throughout the user’s entire life cycle.

“Fine” mainly means that we can collect more comprehensive attributes and finer granularity, so that the collected data is more comprehensive and of better quality, and can better meet our more refined analysis needs and facilitate the accumulation of data assets.

“Time”, mainly refers to real time. For example, a user searches for “toothache”, but our data analysis results come out a week later, by which time the user’s toothache may have “recovered”. Shence attaches great importance to real-time performance. Considering technology and cost, Shence will maximize real-time performance to ensure the timeliness of data analysis and application. At present, from data collection to data analysis and display, the divine strategy has reached the level of minutes.

The functions of the DATA acquisition SDK can be summarized in three aspects:

• Data is collected through burial sites
• Transmits the collected data to the specified server
• Ensure the accuracy and completeness of data to the greatest extent

Common way of burying:

• Code buried point
• Full buried point (no buried point)
• Visualization full buried Point (circle selection)

“Code buried”

The code burying point is that in our product, when a user’s behavior occurs, the INTERFACE that calls the SDK triggers an event.

Code burying point advantages:

• Precise control of buried points (convenient and flexible customization of events and attributes)
• Rich data collection
• It can meet the needs of more refined analysis

Code burying disadvantages:

• Burying point cost is relatively large
• Need to be accompanied by the APP version

“Full burial point”

Full buried point is also called no buried point, no code buried point, automatic buried point, no trace buried point, that is, without r & D engineers to write code or write a small amount of code, you can achieve the purpose of automatic (in advance) buried point. The purpose of full buried point is not only to meet the needs of customers’ actual business analysis, but also to reduce the cost of customers’ buried point.

Events that can be collected at full buried point:

• The activation
• The APP launched
• Exit the APP
• APP Page browsing
• Click on the control
• exposure

Activation is the first startup after the user installs the APP. One of the biggest challenges facing Android and iOS in determining activation is how to solve the problem of uninstalling and reinstalling apps. Generally, a device is uniquely identified. For example, Android may use AndroidID, iOS may use IDFA, etc. Activation events typically carry channel information, such as which channel the current user came from.

APP launch, click on the APP icon, or some other way to get the APP running and displayed in the foreground.

Properties to be collected for startup events:

• First boot (first boot after installation)
• Cold start, hot start (recovering from background)
• Start the source

Start-up sources include:

• The user clicks the icon to launch?
• Jump from the Web?
• Jump from small program?
• Jump from another APP?
• Users hit push to wake up?
• Wake up via Deeplink?

A common definition of APP exit is that the APP is not displayed in the front end. This definition can cover most business scenarios, but for player-type apps, this definition is no longer applicable: We put the player in the background, but continue to use the APP, is it quit? Therefore, the definition varies slightly for different business type scenarios.

Attributes of APP exit event collection:

• Duration of APP use
• Exit way

The exit methods include: press the HOME button to let the APP enter the background, forcibly kill the APP, crash the APP, and jump to other apps.

The definition of exposure varies from business to business. For example, if there is a list page, some people think that the list is exposed as long as it is pulled from the server to the APP, while others think that it is exposed only when each item is displayed in the APP.

Suitable for capturing and exposing scenes:

• advertising
• The list of
• Toast.
• Dialog

On Android, pages refer to activities and fragments. For iOS, a page is a UIViewController.

Page browsing event collection properties:

• Page title ($title)
• Page name ($screen_name)
• Forward address ($referrer)
• Custom attributes

For APP, click control is the behavior with the highest frequency of users. Therefore, the most important work of full buried point is to collect control click.

Control click event collection properties:

• Which control is clicked
• What type of control is it
• Which page the control belongs to
• Control to display text information on
• Extend custom properties

By default, the collection mode is full, that is, the click of all controls on all pages is collected by default. However, some clicks are unnecessary for service analysis, which occupies not only bandwidth but also a large amount of storage space. Therefore, click events collected at full buried point also need to provide rich APIS to flexibly configure click events, such as:

• Which pages do not collect $AppClick events
• Which types of controls do not collect $AppClick events
• The $AppClick event for a particular control is not collected
• Add a custom property to the control’s $AppClick event

Advantages of full buried point:

• The cost of embedding is relatively small
• No need to update APP
• Resolve data “backtracking” issues
• Strong dependence on other advanced features (visualization of full buried points, click graphs)

Disadvantages of full buried point:

• Limited functionality is covered
• Service data cannot be automatically collected
• Unable to meet the needs of more refined analysis
• Compatibility issues
• A large amount of data is transferred and resources are wasted

For Android and iOS, the full buried point principle is slightly different. Due to the implementation details more, in this just do a simple introduction, more details can refer to our SDK source code (background reply [SDK source] can be).

For more technical details on iOS Full Embedding, see iOS Full Embedding Solutions.

As for the startup and exit of Android, multi-process issues also need to be considered. The current definition of Android APP startup and exit is as follows:

Startup: if a page is displayed and the time between the previous page and exit exceeds 30 seconds, it is considered as a startup. Exit: if a page enters the background and no new page is displayed within 30 seconds, it is considered to exit once. For Android control click event collection technology is also more, such as AspectJ, ASM, Javassist, AST and other 8 programs, god policy is currently using ASM. For the technical principles of other solutions, please refer to the book Android Full Buried Solution.

“Visual full buried Point”

Visualized full buried point, also known as circle selection, is configured for full buried point in a visual way, such as:

• Sets/modifies the control ID
• Set/Modify control contents
• Sets/modifies the page title of the page to which the control belongs
• Controls which control click events are not collected
• Adding static properties
• Rename the click event name

Visualized full buried point demonstration, can refer to the following figure, more detailed experience, can apply for shence Demo.

The technical principle of visualization of full buried point mainly relies on three concepts:

• screenshots
• ViewTree
• ViewPath

Screenshot: screenshot of the current page of the APP.

ViewTree: Android and iOS pages, like web pages, can be represented by a tree, which we call a ViewTree. In ViewTree, it contains the hierarchical (parent-child) relationship of the control and the visual information of the control, such as the control type, coordinates, length and width, display text, etc.

ViewPath: In a ViewTree, each control has a unique path from the root node to the current node. This path is called the ViewPath.

When the APP is connected to the magic strategy analysis, the SDK will regularly (every second) send the current screenshot of the APP and the corresponding ViewTree information to the server. The front end takes the screenshot as the background and finds the ViewPath of the corresponding control in the ViewTree according to the position of the user’s mouse click. Then according to the element information in the ViewPath to draw the element rectangular box, at the same time through the ViewPath in the system to uniquely identify the current control, and finally with the control click event collection of the ViewPath to match.

The click graph is similar to the full buried point visualization, which is a different application of the above technology. It shows the user’s click on the current page in a more intuitive way.

Faced with so many ways to bury points, which one do we choose?

In the past 5 years, Shence has served more than 1000 enterprise customers. Through in-depth service to customers, we found that there is not a perfect burial site plan, and each of them has its own advantages and disadvantages.

In the face of so many buried point solutions, we can not blindly pursue saving trouble, but can not pursue the “cool” buried point way, the most important is to choose the buried point way that can best meet our needs according to the actual analysis of needs and business scenarios. If multiple solutions can be satisfied, we can go after “easy” and “cool” solutions.

In the past five years, we have also accumulated some experience, take this opportunity to share with you.

Device ID

Before users log in to the product, they need to uniquely identify the current device, that is, the device ID. The device ID is also slightly different for Android and iOS.

1 ID of an iOS device

UDID

The Unique Device Identifier for iOS devices. It consists of a sequence of 40 hexadecimal digits. Since iOS 5, Apple has banned apps from obtaining Uids through code in order to protect users’ privacy. It also announced that it will ban apps from the APP Store if they are found to have acquired the UDID. Therefore, unique identification of the device cannot be guaranteed.

UUID

Universally Unique Identifier. A sequence of 32 hexadecimal digits, connected by small lines, in the format 8-4-4-4-12. If you delete the application and install it again, a new string will be generated. Therefore, the device cannot be uniquely identified.

Mac Address

An identifier used to represent every site on the Internet, in hexadecimal numbers, with a total of six bytes (48 bits). After iOS 7, Mac addresses are also private. If a Mac Address is requested, a fixed value is returned: 02:00:00:00: 00:00:00. Therefore, unique identification of the device cannot be guaranteed.

IDFA

Identifier For Advertising. Suitable for external: such as advertising promotion, exchange and other cross-application user tracking. A sequence of 32-bit hexadecimal numbers in the same format as the UUID. All apps on the same device will get the same value. After iOS 10.0, when users enable restricted AD tracking, the value obtained is: 00000000-0000-0000-0000-000000000000. It is suitable for device ID.

IDFV

Vendor Identifier, which is the abbreviation of Identifier For Vendor, is suitable For analyzing users’ behaviors in the application. It is also a sequence of 32-bit hexadecimal numbers in the same format as the UUID. Each device has the same value in the application belonging to the same Vendor. If the user uninstalls all apps belonging to this Vendor, the value of IDFV will be reset.

Best practices

In IDFA/IDFV + Keychain mode, data is not stored in the SandBox of the APP. Even after the APP is deleted, data is still stored in the Keychain. If the APP is reinstalled, you can also retrieve data from the Keychain. This can effectively solve the problem of uninstallation and re-installation.

2 ID of an Android device

IMEI

The android.permission.READ_PHONE_STATE permission is required. This method is only available for telephone-enabled devices. When Android Q starts, non-system applications or those not in the whitelist cannot obtain IMEI. Not suitable for device ID.

Serial Number

The value can be obtained in the String SerialNumber = android.os.build. SERIAL format. Permissions are not required, some models may return 0123456789ABCDEF, and the latest Android OS may restrict access. Not suitable for device ID.

Mac Address

Required permissions, Android 6.0 will return a fixed value: 02:00:00.00:00:00:00. Not suitable for device ID.

Best practices

Use AndroidID as the device ID. When the device is first booted, a random 64-bit number is generated and saved as a hexadecimal string. It does not require permissions and is common to tablet devices. The disadvantage is that the device will be reset after factory Settings are restored. However, the device meets the actual analysis requirements to the maximum extent. Android Q should also be considered. For the same device, different apps can obtain different AndroidID.

“Time”

In data collection, there are two time-dependent factors:

• Records the timestamp of the event
• Count how long an action lasts

If we take the client time as the criterion, we may have the following two problems:

• Statistical event duration ($event_duration) is invalid and is negative or large
• The occurrence time of the event is unreasonable

Some of you might say, why not use server time? To improve efficiency and reduce data loss, events collected by the client are cached locally and then synchronized after meeting certain policies.

Why don’t you calibrate the client time through the server? Since it is calibration, network request is required, what if there is no network? What about offline apps? What about events or time counts prior to calibration?

Therefore, the above two solutions are not compatible with various scenarios. Here, we present the divine solution.

1. The statistical time

Both Android and iOS have a concept called Boot Time, which is how long the phone has been booted up. It can be understood that there is a timer in the system. Every time the phone starts, the timer starts from zero. It has nothing to do with the time of the client.

// Android SystemClock.elapsedRealtime()

// iOS NSProcessInfo.processInfo.systemUptime

2. The timestamp of the event

We use a “time correction” strategy.

The user triggers an event (such as a login event) at the phone’s time stamp T1, which is then stored in the local cache. The DATA collection SDK starts to synchronize event data (including login event) at the time stamp T2 of the user’s mobile phone, and then the server can obtain the time stamp T2 of the user’s mobile phone when the DATA collection SDK initiates the Request through the Request Header Date of HTTP(S).

If the timestamp at the T2 and the current service error in an acceptable range of T3 (such as the 30 s, this is because the network request also need a certain amount of time), you can show the current user’s phone time stamps are accurate (we assume that the server timestamp is accurate, the vast majority of cases are also accurate). At the same time, the timestamp T1 of the login event is accurate, and the server does not need to do any special processing when receiving the login event.

If there is a large difference between T2 and T3 (for example, T2 is 15:00 and T3 is 16:00 in the figure above), we can confirm that the timestamp T2 of the current user’s mobile phone is inaccurate, that is, it is one hour later than the timestamp of the server (16:00-15:00 = one hour). Thus, we can assume that the timestamp T1 at the time of the login event is also one hour behind the server, that is 14:00 plus one hour should be 15:00.

Therefore, when the server receives an event, it adds an hour to the timestamp T1 of the login event to 15:00, thus achieving the “time correction” effect.

Of course, a “time correction” strategy does not guarantee 100% resolution of all timestamp related issues. For example, or the graph above example, if the timestamp between T1 and T2, the user mobile phone timestamp of man-made changes happened again, so it won’t be able to correct for time was correct, because the timestamp error difference between T2 and T3, and timestamp T1 and the difference between the corresponding time stamp when the server is not equal.

However, in practice, since the DATA acquisition SDK generally synchronizes data at a short fixed interval (such as 15s), the time interval between timestamp T1 and T2 is relatively short, and the possibility of timestamp change itself is relatively small. Even if a change is actually made, The number of events affected is also small (only a few events may be triggered within 15s). Therefore, the “time correction” strategy can solve more than 99% of the problems related to event timestamps.

In addition, since the implementation logic of the “time correction” strategy is handled on the server side and is relatively simple to implement, we won’t go into the implementation details here.

“Firefighter”

For the SDK, the core purpose is to collect data, but there is a premise that it does not affect the normal business of customers. No matter how hard the development and QA tests, there are always unexpected exceptions. At this point, we must have a firefighter, which controls the SDK’s behavior by remotely delivering the configuration, such as shutting down the SDK.

Synchronization Policy

To ensure the integrity and accuracy of data to the maximum extent, the data collected by the SDK is generally cached locally and synchronized after certain conditions are met.

Common data synchronization policies:

• Events are first cached to local SQlite3
• A certain number of events cached locally will be synchronized (100 by default)
• Synchronization at a fixed time interval (15 seconds by default)
• Synchronization of core event occurrence (trackInstallation, login, etc.)
• APP exits and attempts to synchronize
• Handling of too many events cached locally

The purpose of trying to synchronize data when the APP exits is to prevent users from not starting the APP for a long time, not starting the APP again, or uninstalling the APP directly.

Network Policy

Data synchronization requires network requests. Different services and scenarios have different requirements on network policies. For example, services with high user experience requirements may require data synchronization only over WiFi networks. Therefore, the SDK needs to support flexible configuration of network policies to meet different requirements. By default, the SDK synchronizes data on 3G, 4G, 5G, and WiFi networks.

The SDK supports the following network configurations:

• NetworkType.TYPE_2G
• NetworkType.TYPE_3G
• NetworkType.TYPE_4G
• NetworkType.TYPE_5G
• NetworkType.TYPE_WIF
• NetworkType.TYPE_ALL
• NetworkType.TYPE_NONE

“Through H5 and APP”

In recent years, the mixed development of iOS is becoming more and more popular, and the need to get through APP and H5 is becoming more and more urgent. What is the connection between APP and H5? The so-called “get through” means that after H5 integrates the JavaScript data acquisition SDK, the events triggered by H5 are not synchronized to the server directly, but are first sent to the APP data acquisition SDK, which is processed by the APP data acquisition SDK for secondary processing before being synchronized to the local cache.

Why should APP connect with H5?

Mainly from the following perspectives.

1. Data loss rate

In the industry, the loss rate of data collected by APP terminal is generally around 1%, while that collected by H5 is generally around 5% (mainly due to cache, network or page switching etc.). Therefore, if APP and H5 are connected, all events triggered by H5 can be first sent to the APP end data acquisition SDK, which will be processed by the APP end and then merged into the local cache. After meeting the specific policy, data synchronization can be carried out to reduce the data loss rate from 5% to about 1%.

2. Data accuracy

As is known to all, H5 cannot directly obtain device-related information and can only obtain limited information by parsing the UserAgent value. However, parsing the UserAgent value may cause at least two problems:

1) Some information, such as the version number of the application, cannot be obtained by parsing the UserAgent value.

2) Some information can be obtained by parsing the UserAgent value, but the content may be incorrect.

If APP and H5 are connected, the data acquisition SDK on the APP side can supplement these information to ensure the accuracy and integrity of the event information.

3. The user id

If users use our products before registering or logging in to the APP end, we generally use anonymous IDS to identify users. However, the rules for identifying anonymous users in APP and H5 are different (iOS generally uses IDFA or IDFV, while H5 generally uses cookies), which will lead to one user using our product, resulting in two anonymous users. If APP gets through to H5, the two anonymous ids can be normalized (the anonymous ID of APP end shall prevail).

There are many and complex scenarios to get through. The following common scenarios need to be considered:

• Their H5
• Third Party H5 (non-Strategic customers)
• Third party H5 (Shence Customers)

Article source: Shence Technology Community