Author: Front-end team at Shopee Games

Abstract

The Shopee Games team is committed to enriching the interaction and entertainment in Shopee e-commerce, so that users can get more pleasure after shopping. Meanwhile, Games can also bring continuous active users and more coupon distribution channels for Shopee. In this context, from the beginning of the game, we wanted the game to be lightweight and fast to iterate, to continue to provide users with a variety of game experiences, while not causing a big impact on the size of Shopee App. Therefore, we need to choose the right game engine and build a toolchain that works for Shopee Games.

This article will describe how the Shopee Games team chose the game engine, how to expand the game engine to improve production efficiency, and how to combine the game development process with the mature front-end engineering system to achieve the standardization of Games and improve the quality of research and development. Shopee Games is a game embedded in the Shopee App, so the experience summarized in this article will be of certain significance to business teams with the same needs for embedded Games.

1. Selection of game engine

Shopee Games currently focuses on casual Games. In order to reduce the impact on the volume of Shopee App, the selection of technology will prefer H5 Games. When choosing an H5 game engine, we mainly consider the following factors:

• 2D or 3D games?
• Is it development-friendly? This includes support for TypeScript, good documentation, and a development-led process.
• What about performance and compatibility?
• Is the official tool chain complete?
• Open source?
• Are there successful games?
• Is there any official customer service support?
• Is there any official update?

First of all, our casual games are mostly 2D games, so we’ll focus on a game engine designed for 2D. Although the 3D engine can achieve 2D effects through orthogonal perspective, the rendering performance and lightness are not as good as the specialized 2D engine, so we first exclude the 3D oriented engine, such as Unity3D, LayaBox, three.js, Babylon. 2D engines include lufyLegend. Js and Cocos2D-JS, Egret and Cocos Creator, and Phaser/Pixi and CreateJS.

Then, in terms of sustainability and performance, we can first exclude the older lufyLegend. Js and Cocos2D-JS. CreateJS isn’t really a complete game engine, it’s more of a streamlined rendering engine, and it lacks an overall toolset to support large games.

So, finally, let’s focus on Egret, Cocos Creator, and Phaser. The Phaser rendering engine is the Pixi, and phasers represent both later.

All three game engines support TypeScript and WebGL, and the performance differences are not significant. For the Shopee Games team, Egret has major advantages:

• Egret supports The Canvas mode, so some of the lower-end mobile users in Southeast Asia can run our games;
• Egret’s philosophy is for developers, and our team has strong r&d capabilities. Being developer-oriented makes the game better.
• On the tool chain, Egret has its own keel animation and editor, perfect for our game development.

So, to sum up, we chose Egret as our main engine, and built on Egret’s ecosystem to continuously optimize and build toolchains that make game development more efficient.

2. Egret engine optimization and public libraries

2.1 Egret engine optimization

Egret Engine is an open source game Engine developed by Egret Times that complies with HTML5 standards, including 2D/3D rendering core, EUI system, audio management, resource management and other commonly used modules of game engines.

At present, we have developed four games including Shopee Candy, Shopee Pet, Shopee Fruit and Shopee Link with Egret Engine. During the development and iteration of the project, we found some problems with the official Engine. Business requirements and performance standards cannot be fully met. So we did a custom development of the Egret engine, called the Custom Engine.

2.1.1 Performance Reporting

Report game performance metrics such as FPS, DrawCall, First Paint, GPU Size, etc.

The reporting process is as follows:

By analyzing reported game performance data, we can better analyze performance bottlenecks and focus on improving game performance. Detailed performance indicators are as follows:

2.1.2 Performance Optimization

We’ve made some performance optimizations based on the official engine to help developers improve game performance.

The static graph

In the development process, the scatter graph is combined into a large graph atlas to achieve the purpose of reducing DrawCall.

Dynamic graph

Maps are dynamically merged into one large map while the project is running. When a tile is rendered, the dynamic blending system automatically detects whether the tile has been incorporated into the image collection. If not, and the map meets the conditions for a dynamic map, the map is merged into the map set.

A dynamic composite is a rendering order that selects which tiles to merge into a larger image, ensuring that adjacent drawcalls can be merged into one DrawCall.

Similar to the principle of the previous static combination, the combination texture is used instead of the broken texture to reduce DrawCall. The biggest benefit of dynamic combination is to improve some scenarios that can not be static combination in advance, such as the user’s dress up.

Node order adjustment

The underlying performance optimization of the engine is to ensure that the same texture is rendered in the same order. For example, if the original order of addChild is img1>text>img1, the engine automatically optimizes img1>img1>text to reduce DrawCall.

The DrawCall optimization tool is enabled

Start benchmark.init (null,null,true); Or benchmark.Optimizedc is set to true.

2.1.3 Slim engine

The official engine includes all modules by default, some of which are not used in our actual project. Therefore, in order to reduce the engine pack size, you need to eliminate code that is not needed, such as:

• Native code;
• The Runtime code;
• WX and other small game side compatible code;
• KTX texture-related code;
• ETC Loader code.

Comparison before and after modification:

As a result, the front-end JS load decreased by 16KB, or about 7%. Although this volume may seem small, a small amount of volume optimization can be valuable in some areas where the network is poor.

2.1.4 Bug fixes

For some engine level bugs encountered in the project, we need to fix them ourselves in many cases because the engine official may not update them in time. For example: iOS 14/15 rendering lag, Dragon Bone library rendering, network and sound issues, etc. We were honored to contribute code to help the Egret team solve the iOS 14/15 lag problem.

2.1.5 API enhancements

Some uses of the official engine are cumbersome and not friendly, such as setting the height of the node. Therefore, we have extended the convenient AND fast API on the basis of the official engine for everyone to use.

2.2 public library

In order to improve development efficiency and avoid duplication of wheels, we developed a common library based on the optimized Egret engine to encapsulate common tool classes, common modules, common UI components and so on.

2.2.1 tool library

We’ve wrapped up a few libraries of tools commonly used in games:

• SoundUtil: Music player tools that support sound/background music playback, pausing, and multiples.
• DragonUtil: Keel tool class, responsible for the creation/destruction of keel animation, hidden keel creation details, simplify the use of keel animation;
• ResUtil: Game resource management class, easy for developers to load/release game resources;
• SmartEvent: encapsulated message notification library, easy to use, easy to decouple between modules, including custom events /UI events listening and removal;

The purpose of encapsulating toolsets is to ease development and prevent different teams from reinventing the wheel. It has been used in four Egret Games by Shopee Games, saving more than 2 weeks of manpower on average.

2.2.2 Basic UI Components

We extended Egret base components and provided lifecycle hook functions to make development easier and more efficient. At the same time, it provides some common components of various projects, such as: share interface/friends interface/little monster pop-ups and other public UI components.

Egret base component extensions

We provide lifecycle hook functions for UI components to be used by the game business, so that developers can implement each UI class without having to implement event listening and removal separately. At the same time, built-in event management also avoids memory leaks that developers may have caused by development omissions.

The specific hook functions are as follows:

2.3 Customization Engine Synchronization update

As more and more changes are made to the custom engine, the question is: if the official engine is updated, how can we quickly merge the official engine version?

The scheme adopted here is git dual Remote scheme, and the flow chart is as follows:

The detailed steps are as follows:

• For convenience, we defined the open source Egret engine library as A, and our own customized engine repository as B.
• Clone B with Git, pull and modify project B;
• Through git remote add A repository and Git fetch A, add A remote and obtain A’s repository information;
• Suppose B’s development branch is dev, switch to that branch;

Assuming that what we need to merge is A tag of A, such as V5.4.0, use Git Merge V5.4.0 — Allow-suggested -histories to force the merge.

Due to the need to synchronize the source framework project code, we are limited in our changes, otherwise there will be repeated work with each merge:

• Try not to rename or delete the original file, or change the function and variable names in the code;
• If you need to extend the functionality of a class, try to use the form of prototype chain extension;
• Custom internal utility classes can be internally defined, as long as they do not have the same name.
• In line code as far as possible to use increased mode, as far as possible not to change the original code;
• Some library code will add a lot of channel-compatible code, which we can reduce appropriately, and merge will be based on the mechanism of analyzing changes from a common ancestor, so there will not be diff every time.

With the above solution, we can achieve fast synchronization between the official warehouse and the custom engine.

3. Game r&d engineering

While the Egret engine meets Shopee Games’ basic business needs, there are a number of tools available to meet developers’ needs. However, some of the pain points we encountered with the Egret engine were:

• Lack of module concepts: the default TypeScript compiler does not support the top-level import and export files, and all compiled files are considered globally visible, leading to variable contamination and security issues.
• NPM cannot be used: Package. json files are not supported in the root directory of the service project, and modular third-party libraries are not supported.
• Lack of engineering solutions: no engineering solutions are provided, such as code review, unit testing, etc., and the project cannot easily access the conventional Web front-end engineering solutions;
• The deployment process is complex: the code compilation tool relies on the official tool, does not provide a command line version, and cannot be deployed independently on the server.

It was clear that the Egret project could not meet our engineering requirements. Even though the Web front-end engineering technology is mature today, we are still in the Stone Age, so we decided to engineer the Egret front-end.

3.1 Egret front-end engineering

3.1.1 Support the root directory package.json

Package. json file can be said to be a necessary file for front-end projects at present. Egret engine started relatively early, when front-end engineering was not so mature, and the construction of Egret engine was an official construction system written by itself.

Without support for package.json files in the root directory, many things are hard to do, but Egret engine build tools are written in JS and open source like the engine code.

The reason Egret projects don’t support package.json in the root directory is that Egret builds distinguish between project and library projects by determining the presence of package.json in the root directory, resulting in different build flows and different artifacts.

In order to minimize changes and support the existence of package.json in the root directory of the project, we changed the judgment of building the project to the value of the custom field in package.json to distinguish whether it is a project or not.

Support for package.json in the root directory makes subsequent engineering changes easier.

3.1.2 Engine NPM package

The official construction relies on the construction tool on the local machine. Every deployment and release needs to be uploaded to the server after the local construction is completed, which is not consistent with the deployment specification and process of Shopee business, and seriously hinders the rapid iterative pace of the project.

To enable the build to support standalone deployment on the server, we adapted and packaged the code for the customization engine and published it as an NPM package, changing the project dependencies from a native build tool to an NPM package.

  "dependencies": {
    "@egret-engine/egret-core": 1.6.2 - alpha. "1",}Copy the code

The NPM package consists of two main parts:

• Build directory: library files related to the engine;
• Tools directory: build and compile related tools.

Publishing to NPM not only makes the compilation and execution of the project out of the local environment, but also makes it easier to do project versioning. But it’s not enough to just publish as an NPM package, we also need to package and build with the following Webpack to achieve our goal.

3.1.3 Webpack package build

To support modular compilation and standalone deployment on the server, we selected a mature Webpack build solution to plug into the Egret project.

Before retrofitting an Egret project build, you first need to analyze the Egret project dependencies and build artifacts:

• *.js: Code build artifacts.
• *.ts: TypeScript business code files.
• res: Project resource file. For example, images, audio files, and JSON files.
• egret libs: Egret project depends on modules, which are associated JS library files.
• *.exml: Egret specific tag language file type used for UI layout and compiled into JS and JSON files.

The official build is similar to gulp, where each task is executed in a certain order. Although there is also an official way to customize task plug-ins for developers to customize the construction process, this requires developers to develop again, which is quite labor-intensive.

Exml file type is unique to Egret engine. Currently, there is no relevant parsing and compilation tool in front-end ecosystem. There was no need to re-engineer the wheel for the RES file handling, so we stuck with the official tool and packaged it in @egret-egine/egret-core/ Tools as a dependency on the build tool.

With Egret Libs dependency processing and *.TS code compilation, we were able to find a better solution on the front-end ecology and use it as needed.

Egret projects are packaged with Webpack and build dependencies are sourced from NPM so that builds can be deployed directly on the server out of the local environment. And the products are also consistent with the official packaging products, so as to achieve good compatibility.

3.1.4 Engineering Configuration

After the above modification, Egret project is not much different from normal Web front-end engineering. Mature Web front-end engineering solutions can be well practiced in our project, which can not only be deployed independently on the server, but also easily access quality control tools such as ESLint, JEST, etc. Improve code quality.

3.2 Egret – Webpack – CLI implementation

At the beginning of the project, we built the project scaffolding template based on Egret and Webpack based on business and engineering requirements. However, when creating new projects and demo projects, it is still necessary to come down from the Clone template repository and perform some manual configuration according to the project. In the current use, the problem is not big, but it is still tedious, there may be some omitted configuration, new projects can not be done out of the box. So the development of scaffolding tools, can quickly generate the corresponding template project.

3.2.1 CLI

General scaffolding tools are divided into CLI and Template two parts. Scaffolding template content is not placed in the same repository with the CLI, but in different repositories for management and iteration. Separated, the two parts can be maintained independently and do not interfere with each other. For template configuration or dependency update, you only need to update the project template, which does not affect the CLI and cause package redelivery.

Following other scaffolding ideas, templates are published to remote repositories as independent resources, and then downloaded from CLI tools during operation. Interaction information through CLI is used as input meta information to render project templates.

After EGREt -CLI is executed, the terminal can generate the corresponding EGRET project according to the interaction information.

3.2.2 the Template

Because we need to respond to different requirements and there are many configurations related to business, the service configurations of the template vary greatly, so a unified template cannot be completely achieved. At the same time, in order to ensure that there is no redundant configuration in a template, we make a distinction and mainly provide four project templates: Base, Standard, Shopee and Native.

When you need to do a small demo, you can directly use the Base template, relatively simple; If you need to explore business-related features, choose the Shopee template. For new projects, use native templates directly.

3.3 Final Egret game development process

Egret project is in line with conventional Web front-end engineering, which not only solves the development pain points and meets the engineering needs, but also enables us to formally enter the industrial age from the Stone Age. From development to deployment, we have excellent tools to assist the implementation, which improves the code quality and development efficiency. New students can also handle the project well.

Mature Web front-end engineering not only helps us expand our business, but also gives the project more possibilities. Some features that were easy to implement in the front end but difficult to implement in the original game engine, such as dynamic logic code loading, multi-page applications, Egret+React hybrid pages, etc., were also well implemented in our project.

4. More R&D issues

4.1 Background of iOS Audit Problems

In the early days of Shopee Games, when users and visitors were small, Apple didn’t focus on the HTML5 version of Shopee Games. However, as Shopee’s business grew and users and visits continued to increase, In early 2021, Apple proposed HTML5 games built into Shopee, Shopee App is considered to be in violation of Article 4.7 of App Store Review Guidelines.

Apple’s censorship terms read like this:

4.7.1 Software offered under this rule must: -be free or purchased using in-app purchase; - only use capabilities available in a standard WebKit view (e.g. it must open and run natively in Safari without modifications or additional software); and use WebKit and JavaScript Core to run third-party software and should not attempt to extend or expose native platform APIs to third-party software; - be offered by developers that have joined the Apple Developer Program and signed the Apple Developer Program License Agreement; - not provide access to real money gaming, lotteries, or charitable donations; - adhere to the terms of these App Store Review Guidelines (e.g. do not include objectionable content); and - not offer digital goods or services for sale.Copy the code

To sum up, it contains the following requirements:

• All free H5 games or apple Pay;
• Only WebKit functions are used, and extensions such as JSBridge are not allowed.
• H5 developers need to join Apple’s Developer Program;
• Money gambling is not allowed, and content is subject to other censorship rules.

However, Shopee Games is embedded in Shopee App, which naturally needs to be deeply combined with Shopee and inevitably involves JSBridge (such as login and jump shop). Shopee Games also uses Shopee Coins as currency in its Games. Shopee Coins are not earned through Apple Pay, but are accumulated through e-commerce purchases. From both perspectives, Shopee Games’ use of HTML5 is bound to violate apple’s censorship rules.

We took the official Egret engine advice and changed the technical architecture of Shopee Games on iOS from HTML5 to Native, using Egret Native Runtime. The official Egret tool supports one-click iOS project generation and release of the corresponding App installation package, which is suitable for indie games. However, Shopee Games needs to be embedded in Shopee App, which is not an independent game App, so it cannot simply use the official one-click publishing mechanism. We need to further study the principle of Egret Native to integrate with Shopee App.

4.2 Egret Native Principle

Analyzing the iOS template project created with Egret Native, we found that:

• EgretNative Runtime relies on a suite of system libraries and a stand-alone package called Libegretnativeios. a, where the core logic and network functionality are located. Since Egret Native is not open source, the implementation is not known from the template project;

• The assets folder in the App root directory is fixed as the resource directory. The generated files of H5 version need to be fixed here, and only one game is supported here.
• With the libEgretNativeIOS library, you can create the corresponding EgretNativeIOS instance and corresponding View.

While Egret Native is not open source, further Egret Native can be discovered in conjunction with emulator breakpoint debugging analysis, according to the official documentation.

The Egret Native game architecture consists of three layers: the front-end game layer, the Egret Native Runtime and the iOS Native layer.

• The front-end game layer is consistent with the H5 version of the file content;
• Egret Native Runtime is the core adaptation layer, which uses JSCore to parse JS files in the game package and build a JSBridge to realize communication between JS and Native. The Runtime rendering approach differs from that of the Web. Egret Native has a complement of JS Polyfill and JS Engine for Native and does not implement all browser functionality at the Runtime level.
• The iOS Native layer manages the Egret Native Runtime lifecycle and manages views.

To launch an Egret Native game, initialize an Egret Native instance from the iOS Native layer, create a corresponding Game View, and mount it to the main screen. Then Egret Native initializes the JS engine, binds JSBridge, reads the game resources of the front-end game layer, parses HTML and JS, calls the OpenGL interface, and finally displays the game screen.

4.3 Egret Native and Shopee App integration

From the template engineering point of view, the way to embed EgretNative game into Shopee App is relatively clear. LibEgretNativeIOS. A and other necessary dependency libraries are added to Shopee App project. Then store the game package in assets directory, and finally bind the necessary Shopee JSBridge for the game logic to realize login, payment, jump and other operations, the whole combination work is complete.

However, in the final implementation process, some deep-seated problems are found, which need to be solved or avoided through the business side. These issues include:

1) The template project does not support multiple games

Shopee Games contains multiple Games, and the assets directory can only store the resources of one game. And EgretNative Runtime, namely the above libegretnativeios. a is not open source, so targeted modifications cannot be made.

Finally, we found in the official Hot Update Solution that Egret Native can set the preload directory path, and the resources in the Preload directory have a higher priority than assets.

Therefore, we can build multiple games into Shopee App, store them in a separate directory, and set the preload path as the path of the corresponding game before launching Egret Native. Egret Native will start the game by reading preload resources first, ignoring subsequent assets.

2) The network cache file increases indefinitely

Egret Native Runtime caches network requests, but there is no Egret Native Runtime cleanup mechanism, resulting in a local cache that grows exponentially as the game downloads more resources. This part needs to be completed at the level of Shopee App iOS Native logic.

3) Some third-party libraries have naming conflicts

Libegretnativeios. a comes with some third-party libraries, such as SocketRocket, and Shopee App also introduces this library. Neither party handles the alias of this library, resulting in a naming conflict compilation failure. Because libegretnativeio. a cannot be modified, SocketRocket can only be modified at the business level of Shopee App.

4) Memory leak at Egret Native

Since Egret Native Runtime needs to be repeatedly started and destroyed in Shopee App, any improper handling of objects and textures by Egret Native Runtime will cause memory leaks. With Egret Native indie games, this problem doesn’t exist because every time you shut down the game, the entire App is destroyed. This problem has been fed back to Egret official, but as the official team will not deal with it specifically for our use scenario, there is no progress at present. Fortunately, this part of the memory leak is small enough not to be a major problem for now.

By solving the above problems, we finally realized the integration of Egret Native and Shopee App, which enabled us to run a number of self-developed games on the Shopee App. From apple’s review point of view, our approach is separate from Webkit, and all code logic is built into the installation package of the review, which fully meets the requirements of the review. Therefore, after the scheme was launched, Shopee App and Shopee Games successfully passed the review of App Store.

4.4 Future Planning

Egret Native has been integrated into the Shopee App, but after six months of continuous operation, we still have some problems with Egret Native:

• Runtime is not open source and has unsolvable problems;
• The Runtime only supports Egret games, not other game engines. In particular, the Egret engine is not yet the best choice for 3D game development.

Therefore, we are working on a more long-term solution — a homegrown Native Runtime that supports multiple H5 game engines simultaneously.

This solution adopts the overall structure similar to wechat mini-game, but further encapsulation will be made in JS level, which is convenient for quick access of all game teams in Shopee. Technically speaking, Native JS engine exposes WebGL standard aligned interface and necessary BOM interface to JS side, so that ordinary H5 game engine can run seamlessly on browser and our own Runtime.

The benefits of this scheme are:

• Good compatibility, can support multiple game engines at the same time;
• Convenient stock H5 game transfer;
• Leveraging the proven development tool chains of each engine.

However, there are corresponding disadvantages:

• OpenGL support is being phased out in versions after iOS 12;
• Compatibility with multiple engines makes it difficult to use new graphics standards such as Metal, WebGL2, Vulkan.

Now that iOS has reached iteration 15, OpenGL and WebGL are still running smoothly, and the WebGL standard cannot be bypassed to support multiple H5 game engines, so these disadvantages have to be accepted for the time being. In addition, wechat mini games will also face the same problem. I believe that there will be an official Apple migration scheme in the future, which may encapsulate the OpenGL interface in the upper layer of Metal.

At present, this program is still under pre-research and development, and it is expected to be realized and fully used in 2022. In the future, not only Shopee Games, but we also hope to reuse this 3D/GPU rendering scheme in more e-stores to create more colorful interactive gameplay for users.

4. To summarize

Comparing several H5 game engines, Shopee Games chose AN Egret engine that was more suited to the business characteristics and team talent characteristics.

During our long-term business development operations, we customized the Egret engine to better support business requirements, including bug fixes and common library modifications.

While optimizing the engine, in order to keep in sync with the official repository, we made use of the feature of Git multi-remote repository to realize the code merge of dual repository.

Further, in order to reuse the mature front-end Webpack construction system and CI/CD process, we developed Egret-Webpack-CLI and changed Egret game from the original stand-alone and local packaging mode to server Webpack packaging. This makes it easy to reuse a large number of excellent front-end NPM libraries. These innovations have brought significant improvements to Shopee Games’ research and development.

We encountered some difficulties in the iOS review, and finally successfully implemented the Native version of the game through in-depth integration of Egret Native and Shopee apps and passed the App Store review.