❝
This article comes from Huang Yu, the user growth front end team of Tao Department
Background and pain points
In Amoy system front end, we do not have a simple and efficient AB experimental access scheme. In the past, most front ends received AB requirements, the only choice is to call the library function provided by the client to achieve, we found some pain points:
- First, front-end developers need to spend a lot of time dealing with the logic of AB;
- In addition, the scope of the client experiment is only limited to the end, and can not cover all the front-end scenarios, such as outsourcing, small programs or PC, etc.
- The precision of data is insufficient. Both server-side experiments and client-side experiments are page dimensions, while the business side often needs to look at more fine-grained indicators, such as the click rate of a certain module on the page. The front-end has natural advantages in data collection.
Therefore, to create a simple and efficient standardized AB experimental link that serves Amoy department and even the front end of the group is our original intention to do this. This paper describes the design and principle of front-end AB experiment.
Design ideas
What exactly does a complete AB lab link require? Create experiments, engineering access, diversion, experimental data reflux… Well, yes, I could list a lot more, but if they’re not organized and structured, they’re meaningless, let alone architectural. So, let’s try to group these discrete elements.
A complete AB experimental link can actually be divided into two parts, the first part isExperimental configuration linkAs the name implies, it contains all the operations associated with experimental configuration, configuration creation, saving, grouping, configuration push or delivery, and real-time streaming using configuration. The second part isExperimental data link, it contains experimental data dyeing, filtering, calculation, display (experimental report). We put these two parts together into a closed loop, which is the core of the AB experiment we designed for the front end.
These two data models are described below.
Experimental configuration model
We will experiment with configuration toapplicationCombine for the dimensions and push to the CDN. The front-end runtime reads the configuration through the JSSDK, completes the shunt and renders the corresponding business components.
Application we can understand as a front-end engineering, a small program can be an application, or hand wash a certain activity page (such as gold) can also be an application. The application contains a scene, a separate traffic entry we call a sceneFor example, the home page of an application is a scene, and the shopping guide link is also a scene. The scene is also the underlying triage model, which will be introduced in detail in the next chapter. Here we only need to know that the experiment we created is directly related to the scene.
Experimental data model
From the figure above we can conclude that,A business is a collection of data metricsAnd experiments under the same business can create and associate all indicators under this business domain. These associated indicators will eventually be reflected in the experimental report through data collection and calculation.
Take a practical example, the Amoy gold coin business in Amoy gold coin home page (scene) we want to run an experiment, AB marketing module which is more efficient? So the exposure UV and click UV of the marketing module are the metrics we need to create and associate. This metric also exists in a data set within the business domain.At this point, some people may wonder why there is a distinction between the two concepts if gold mining is both application and business? Why not an application that delivers both configuration and composite metrics? The reason is very simple,Because business can cross applicationsFor example, due to the outstanding performance of a certain business of Taobao, it decided to expand and go online in taobao special price version App. So this business has two applications, and one is a special edition of hand wash, here configuration distribution to distinguish, but many business indicators are the same, in this case, one business two applications is a relatively optimized solution.
Design scheme
The design of front-end AB experimental link is like a cloze of the above two models. Starting from the two major directions of platform side and running time, we arranged the elements and their relations of the above model one by one, and formed a closed loop in series.
AB platform
For businesses entering AB platform for the first time, we need to register the relevant workspace
- Application, used to manage experimental configuration, that is, configuration integration and push
- Business, used to manage experimental data indicators
- Scenario, triage model (described in detail in later chapters)
After creating the workspace, we started the core experiment to create links, which included reading the basic information of the experiment and setting buckets (traffic distribution). In order to further reduce the threshold of experimental access, the platform side will automatically generate front-end access code after the flow allocation step, so that front-end developers can quickly access the experiment.
Next comes the process of creating metrics and associating them with experiments (refer back to the experiment Data Model section). Then comes the experimental release. Before the official release, there is a beta release, which can be understood as an “informal” release before the experiment goes live. Developers and business parties can fully verify the experimental flow, data and relevant business logic after the beta release before the official release. Of course, experiments can be published multiple times, meaning that in the middle of the experiment the business side can go back to the traffic allocation step, re-regulate the traffic ratio, and re-publish the experiment.
The runtime
JSSDK
Recall the position of the JSSDK in the experimental configuration model diagram above, which runs in the front end project,Read experimental configurations, stream in real time, and return components to render based on the stream results. Let’s review the experimental data model again, what does THE JSSDK do with the data,The answer is experimental data reporting. First, it needs to report triage results, and second, it needs to report the index data associated with the experiment, that is, the experiment data that the business needs to see. It’s clear what we want to do with JSSDK, so let’s talk about our specific design and extension plan.
The following figure illustrates how the JSSDK relates to front-end engineering and the AB platform at runtime. Starting from the front-end engineering, we introduced a business AB experimental component in the project, which is dynamically generated by AB platform according to user configuration. As a connector between (AB experimental related) business component and page (or parent container), its core work is to read the parameters required by AB experiment and pass them into SDK.This triggers the entire AB experiment logic of the SDK.
Looking at the JSSDK part (blue), first of all, we split the JSSDK into two packages:
- One is the Core package, which encapsulates the common Core logic such as experimental configuration read and cache policy, experimental cycle control and shunting algorithm.
- The other is Coupler, which connects to a specific DSL project. As shown in the figure, it is like a transfer station for the AB experimental flow. It implements a set of interface functions, namely request, cache, and cookie resolution (shunt factor) in a specific DSL environment (e.g. React). These interface functions and experimental parameters are passed through to Core. The purpose of this design is to completely decouple Core from the front-end DSL, which greatly increases the scalability of JSSDK.
Get the experimental parameters, and the interface function, the Core of work to do is to get the experimental configuration, as the Core will not go directly to request the experimental configuration, but the trigger version control strategy, the strategy is primarily to check if the version number of the remote experiment configuration updates, if any update to request configuration, otherwise you will read the local cache configuration. The configuration of the local cache is requested and cached from the CDN when the user first triggers the experiment, and the cache is re-requested and updated with each subsequent version update. After getting the experimental configuration, Core will confirm whether the current experimental cycle (AB platform side configuration) is in advance. After passing the verification, it will formally trigger the shunting algorithm (see the next section), and then return the shunting result to Coupler. Coupler will then judge which corresponding business component should be displayed according to the triage result, and it will report the triage result to the platform. Users can see the triage data in the real-time data report of the experiment at this time, and technical students can confirm whether the experiment is triggered normally through the real-time data. In addition, the embedded component will make a layer of encapsulation for the hit business component, which will pass in the embedded reporting method that can be called by the business component. The specific calls have been generated when we create experiments on the AB platform, and the front-end students need to deploy these codes for reporting experimental indicators to the corresponding business. This part of buried point data is T+1, and the business side can see the corresponding experimental report on the platform side and analyze the experimental results.
Extension scheme
Front-end DSLS can be said to be a hundred schools of thought, in order to cover all front-end scenarios, we have carefully considered the scalability of JSSDK in the design. This is why we split JSSDK into Core and Coupler packages in the Rax 1.0 scheme in the last section. Our idea is that Core encapsulates the Core logic of AB experiment, does not rely on any front-end DSL, and introduces a “connector” package between Core and front-end engineering to connect the whole link. As shown in the figure below, we can use this architecture to scale very cheaply to React, applets, Node FaaS, and more. It also has good maintainability.
In fact, as mentioned in the previous section, how do we completely decouple Core from these DSLSThe interface specification“, and then implements a set of interface functions based on this specification in the “Coupler” package. When Core calls these functions in the execution of a piece of logic, it doesn’t care which DSL API it’s using underneath. For example, React and applets have completely different apis for localStorage operations, so we implemented a set of functions for localStorage processing in React and applets according to interface specifications, and passed them to Core transparently.
Shunt model
Orthogonal and mutually exclusive we mentioned scenarios in chapter 1 when we talked about experimental traffic models. We defined a single traffic entry as a scenario. For example, the home page of XX application can be a scene, now we might as well expand this example, the home page of XX may run multiple AB experiments at the same time:
- In experiment 1, the red envelope interest layer has two styles, AB, and the observation index is the click rate of the two layers;
- In experiment 2, the commodity module of the page header has two styles of AB, and the observation index is the click rate of the module.
- Experiment three, the operation of the marketing banner has AB two designs, through different interest points, the observation index is the click rate of the banner;
Now, the problem is that experiment 2 and Experiment 3 are both modules on the page, and we want them to run at the same time, but we don’t want them to interfere with each other, meaning that the traffic into experiment 2 and experiment 3 must be mutually exclusive. How do you do that? For us,The scene is not only a traffic inlet, but also a diversion modelThe experiment is not placed in disorder in the scene, but throughlayer(layer) for specification. One way to think about it is,The scene is a vertical container, while the layer is a horizontal container, and experiments are placed in different layers according to certain rules. As shown in the figure below, experiment 1 monopolizes the first layer and has an orthogonal relationship with experiment 2 and experiment 3 of the next layer, that is, the flow into experiment 1 will also enter experiment 2 or Experiment 3. We put experiment 2 and Experiment 3 in the same layer, because we want the flow into experiment 2 not to overlap with the flow into experiment 3. This description can be summed up simply as,In one scenario, the experimental flows between layers are orthogonal, and the experimental flows within layers are mutually exclusive.
The experimental push all
In the model diagram above, we also see a special Layer, Launch Layer, which is used to put the experiment groups that have been promoted. Through online validation experiments, for example, one effect of group A is better than group B, the user in the platform side will be pushed to the whole group A, this time the scene internal structure change, namely the group A will be from the original experiment 1 layer in this special Launch layer, this time the scene all triggers in experiment 1 traffic will no longer perform triage algorithm, Instead, the front-end component corresponding to group A is returned directly.
JSSDK Traffic splitting Rules of the JSSDK follow the traffic splitting model mentioned above. In the front-end AB experiment, the traffic splitting factor is the CNA field in the cookie of the user’s browser, that is, the USER’s Web device ID. We use this field to uniquely identify a user. Its advantage is that it can cover various scenarios, such as end-to-end, end-to-end, wireless and PC, etc. The disadvantage is that when the same user accesses front-end applications using different Web devices, different grouping results may be displayed. In addition, the Web device ID cannot be used to determine the user’s population. A shunt factor must have its limitations, and this part of the design needs to be extensible, such as support for future access by user ID, UTDID and even various user-defined shunt factors.
The data flow
Back to data links, in the AB Platform design section we talked about creating data metrics and binding metrics to experiments, in this stepEach newly created data metric is assigned a unique ID (UID) to identify the metricWhen the front-end engineering is running, corresponding components are displayed after shunt and relevant buried points are triggered, as shown in the figure below. In buried point parameters, we will bring log key, buried point type, experimental parameters, namely experimental release ID and group ID. What is the experimental release ID? As mentioned above, in the experimental operation, we allow the business to re-regulate the traffic ratio and publish the experiment, that is to say, an experiment can be published multiple times.Therefore, the experimental release ID can identify that the data collected is a certain release of the current experiment; Of course, we also carry the UID that identifies the metric.
On the platform side, the scheduled data task will be run, which will filter out experimental data from the bottom table. As mentioned above, we will report a log key when the buried point is reported. This key is a key agreed between SDK and data side, which is used to mark this record as the buried point of the experiment. Since a large number of logs are written to our log base table every day, this special log key allows us to quickly filter out experiment-related logs from various logs in our log base table. Then, we classified the data according to the reported parameters. Experimental release ID was used to identify a release of an experiment, UID was used to identify the index, and bucket ID was used to identify the group of the experiment. In this way, we could get the UV and PV corresponding to the index, and the report of AB experiment was also generated.
conclusion
Front-end AB experiment in Amoy department and even the group, or a piece of land has not been fully reclaimed. We iterate a lot, but we rarely verify, or scientifically verify, the true value of an iteration. In the era of the rise of big data, it is far from enough to rely only on “experience” if every iteration cannot finally precipitate data. Front end engineers, from now on we have a place where we can use scientific rigour to experiment, validate iterations with experimental data, and make decisions based on that. Finally, welcome interested friends to further exchange, you can add my wechat MrMarc, looking forward to your insights.
