Pebble, Tencent Interactive’s open source distributed development framework

The author:
Hanwei enterprise

Pebble for building the game world

Vision: A great underlying framework for game servers

In modern game projects, in order for more players to play together, game servers need to host an increasing number of online players. At the same time, to make the game more enjoyable, more and more complex business logic needs to be run on the game server side. Therefore, it is becoming increasingly necessary to have an underlying framework that can also be flexibly extended and easily build server clusters. This allows game programmers to focus on writing fun game logic without worrying about server load.

Pebble means “cornerstone.” Our goal with this framework was to make it robust, reliable, and serve as the basic development framework for the game server. Pebble was born out of the Public Technology R&d Center of Tencent’s game development department, which has been providing underlying technology and services for Tencent games for years. From terminal games to mobile games to video games, the team has been in the forefront of game development technology. Pebble is the culmination of years of gaming technology and support experience.

Values: Ease of use, flexibility, cloud

Pebble was designed with three things in mind: ease of use; Flexibility; The cloud characteristics. To make it easier for more game teams to build a game server, Pebble carefully weighs features, simplifies a library of features most commonly used on the game server, and makes the code run on Linux alone. The Pebble library itself is also a well-layered SDK that allows developers to quickly build a game server using the full functionality of Pebble. You can also extract only the underlying libraries that feature them and use them to enhance your game server programs. To make the game server truly a PaaS cloud, Pebble also prefabricated a complete cluster management solution in the code base, making it easy to set up a game server on any IaaS cloud server. Pebble also works well with Tencent Cloud’s “Game Cloud” solution, enabling any game server to access the cloud with one click, eliminating all installation, deployment, and resource planning, and providing complete statistics and monitoring services.

Business characteristics

Operating environment:

• Pebble runs on Linux and is distributed as a C++ static library + header file.
• Client engine support: Unity3D (C# library), cocos2d-x (C++ library);
• Pebble provides a template project project directory, as well as a “Blade” compilation tool that replaces the traditional Make. (This tool simplifies the tedious process of writing makefiles. But you can still use make or any other compilation tool to build projects using Pebble, since static library files and header files are standard.)

Communications functions

• Pebble supports client-server and server-server communication. Pebble provides libraries that make it easy for mobile clients to communicate with Linux server processes. The Pebble library can also be used to easily communicate between Linux server processes. Pebble itself supports multiple transport and encoding protocols, including TCP and HTTP for transport, Binary and JSON for encoding, and users can customize the protocols supported by the extension.
• The interface to the communication function is provided in the form of RPC: The user writes a function IDL file (in the same format as Thrift, Facebook’s popular open source software), defines a function and its parameters, and Pebble builds the underlying communication code for this function. The user only needs to call the auto-generated function “stub code”, the remote “skeleton code” function is automatically called, and the return value is automatically passed back to the caller. Users only need to fill in the template vacancy of the function “skeleton code”, and then they can realize the function they need.
• Pebble’s RPC, in addition to the standard “request-response” model, also supports “one-way RPC,” which simply means RPC without waiting for a return value, which is useful in some of the “notification, synchronization” scenarios in games and saves a lot of latency. Pebble also supports the ability to broadcast communications: users can simply register a “reverse RPC” skeleton function on the phone (or server) as a notification handler to broadcast data from a Linux server to any number or combination of terminals, both phone and server. The goal of broadcasting is channel management using the API provided by Pebble. The channel data is automatically shared across the cluster, so you can use the entire Pebble server cluster to broadcast without having to have all your mobile clients connect to a specific server at the same time.

Asynchronous frameworks and coroutines

• Pebble itself is an asynchronous, single-process framework that provides very high processing throughput while greatly simplifying concurrent data sharing in complex business logic. Asynchronous development of business logic is also recommended when using Pebble. Pebble itself supports asynchronous development by providing interfaces to handle common signals such as start, reload, and stop services, as well as a timer API for business use.
• While asynchronous code has a performance advantage, it is often less readable than synchronous code because a complete business process can be split into multiple asynchronous callback code, making it difficult to get a complete impression of the process when reading the code. So Pebble introduced the ability of coroutines: for every RPC process a user makes, a coroutine is automatically created. That is, the user can switch out with Yiled() at any time while writing the “skeleton code” content of RPC, and then switch back in with Resume() after the asynchronous processing is complete. In this way, a business logic that needs to be processed multiple times asynchronously can be fully documented in the same function. In addition to being able to use coroutines in RPC, the timer’s callback function also establishes a separate coroutine for business code to use.
• Pebble’s IDL is not just a description language for generating RPC code, it also allows users to “annotate” its functionality. The user can only define a variety of features and configuration items of an object, written in IDL with the syntax of “annotations”, and then these unique logic, will be in the syntax framework of “annotations”, the callback of some user-defined code, thus producing a kaleidoscopic ability. For example, you can use annotations to implement special initializations, or write back data objects to specific storage devices…
• Like frameworks, the Pebble framework also provides some useful miscellaneous features: reading configurations from INI files and overwriting them with command-line arguments or system variables; Provides practical day configuration and printing functions; Setting the process version……

Data caching and persistence

• One of the characteristics of game business development is the large number of complex business data structures. These data structures are often subject to constant revision during game development and operation. At the same time, the data in these data structures generally need to be transmitted between the client side and the server side, and then cached and processed at the server side, and finally written back (persisted) to the database. Pebble makes this process easier by offering a full set of functional apis that allow users to solve communication, caching, and persistence all at once.
• Pebble provides a STD :: Map interface that lets user-defined data structures be buffered into a pool of objects in the form of PUT/GET. The object pool is shared across the cluster, meaning that any process put puts an object and other processes get it.
• To improve performance, Pebble’s object cache pool is “locally cached,” meaning that if you continuously put/get an object in a process, the object is actually cached in the memory of the local machine, which makes for very high read/write performance. The synchronization of objects across multiple machines is handled by Pebble itself under the ultimate consistency principle.
• To simplify object-to-database persistence, Pebble provides inline operations that write objects to the MySQL database and Redis storage. All you need to do is call a Save() function to persist; And you can also set up automatic write back persistence. The association of object fields with the persistent device table structure does not require the user to write SQL code, but can be done with a simple “annotation” on IDL. Pebble also supports persistence of objects to TCAPlus cloud, a more powerful professional game storage device from Tencent. Finally, if you need to support other persistent storage devices, you can use Pebble’s persistence API to write your own storage and read operations. You can also take advantage of IDL’s annotation capabilities to define relationships between objects and persistence structures and associate executing code.

The cluster characteristics

Three working modes

• Since Pebble was designed to build large-scale clusters of game servers, it comes with rich clustering features. Pebble also makes it easy to deploy servers independently, given that games often need to be built quickly on small servers during development and testing. So Pebble designed the standalone features for three models: standalone, Cluster, cloud
• Single-server mode: For game development and functional testing, in this model, the game server is designed to run independently of any software or service. Just get your program to compile and pass, and you can test it directly without even connecting to the database. Pebble in this mode is fully functional, so you don’t have to worry about changing your code later. In this mode, the game server is limited to a single process, and you can copy the entire run file and associated directories to any Linux system to run.
• Cluster mode: Service processes written by the Pebble framework can be easily deployed as a cluster. Nodes in this cluster have load balancing, automatic DISASTER recovery (Dr), and dynamic capacity expansion capabilities. You don’t need to write any code to solve the toughest problems in these clustered systems. You simply use the same code you used in single-server mode, modify the startup command line parameters (or configuration files), and start a few background monitoring processes.
• Cloud mode: In Pebble’s cluster mode, most of the heavy load is already done. But if you need to use hardware resources more optimally, you should use the cloud model. Pebble’s cloud model gives you the ability to isolate and dynamically adjust resources in your cluster, allowing you to make the most of your server resources. At the same time, cloud service providers also provide a perfect statistics and monitoring system on the WEB interface, which can master the operation of the whole service cloud in real time.

Peer-to-peer model: Pebble’s cluster consists of two types of processes. One is a “cluster centric” process that manages and stores the runtime health of the entire cluster. These processes are served by ZooKeeper and do not require user development. Another category are “service processes,” which perform business functions and are compiled by users using Pebble library development. All “service processes” follow the simplest collaboration logic: each process is peer and reports its own status to the cluster center. When a requirement makes a request, it obtains the specific address of the target process from the “cluster center” before making the request. So the Pebble cluster has a great advantage: deployment is very simple — each process only needs to configure its own service port, along with a “central address” to represent the cluster. Once these two addresses are configured, you can form a complex cluster of any size. In addition, because of the existence of “cluster center”, you can also read the real-time status of the whole cluster, and use this information to develop their own cluster monitoring and management system. Pebble provides a “service management” API for developing your cluster management system.

Scalability based on service routing: Pebble encapsulates the functionality of server-side code as a “service,” so server scalability scales according to the request goals of the “service.” In short, Pebble’s “services” can run on multiple servers that can be added or shrunk without affecting the delivery of the “services.” The basic way to do this is because Pebble clusters route service requests: select cluster nodes (servers) that are available and comply with business subscription rules to respond to requests. This process is completely automatic. You can kill any server process in the Pebble cluster and no service requests will be sent to the invalid process. You can simply start a Pebble server process, it will automatically join the cluster you want, and soon service requests will be sent to the new process. In addition, Pebble can also support the choice of multiple service nodes if the business function has specific requirements, such as hashing a value, by sending the hash value to the request API when the service is requested. This greatly simplifies the tedious process of synchronizing the status of online service nodes, calculating the routing rules of specific services, and finally requesting services.

Stateful services The most difficult scalability and disaster recovery problem in a server cluster is operating on stateful service nodes. Because state data can be lost in scaling and disaster recovery, there is a lot of synchronization, landing, and cache management code involved. This code is repeatedly developed across a large number of service clusters, resulting in a large amount of wasted manpower. Pebble’s solution to service state data is to “make the service process as state-independent as possible.” Pebble provides a standard state storage interface for clustered server processes, allowing users to host state data to Pebble so that users don’t have to maintain complex state validity and scalability themselves. This managed state data interface is very easy to use. It is just a MAP interface of STL, but this map is a distributed storage map. It will set up memory cache (available shared memory) locally in the caller, and then automatically drop (persist) to the storage device when the data needs to be modified, cooled, etc. Automatic data synchronization when multiple service nodes share a record. Pebble can configure and switch data between final consistency, weak consistency, and strong consistency to provide different performance outputs. The rise of cloud

Tencent Cloud: Pebble, a distributed computing platform, is a service framework with PaaS cloud capabilities. So we built a distributed computing platform to host the Pebble cluster on top of IaaS, the foundation of Tencent Cloud. The platform provides Pebble clusters with service deployment, resource procurement and isolation, statistical monitoring, automatic disaster recovery, smart scaling, and more. Simply put, you can upload your Pebble service code, and all the capacity design, hardware procurement, deployment, and monitoring will be done by Tencent Cloud’s distributed computing platform. Almost all technical operations, including capacity expansion, fault recovery, and load balancing, are not handled by the user, but you can monitor these events and effects in great detail, and you can intervene to adjust the automation strategy.

As a framework of cloud services, o&M monitoring WEB tools and log services must attach great importance to O&M operations and status monitoring. Because Pebble is a service framework, it can collect very detailed information, including the success rate of calls for each service, latency, dependency path trees, and so on. It can also provide detailed exception alarm policies based on the details of these dynamic statistics. It also provides detailed records and on-site information for troubleshooting during operation. All of these operations, Tencent cloud, are a beautiful WEB interface; It also provides rich RESTful management interfaces, enabling users to customize dedicated O&M tools. In addition to monitoring health information, Pebble provides a logging framework that connects to Tencent Cloud’s distributed logging platform for games, which uses parallel computing on a large number of servers to extract all the information you need from a large number of logs.

As a distributed service cluster, the operation and maintenance of automatic elastic scaling is usually very heavy. With thousands of servers running at the same time, hardware and network failures are common. The number of users fluctuated greatly, and the workload of resource procurement and recovery was also large; Product content changes rapidly, versions are updated, and new environments are deployed frequently — all of which require professional operations teams. However, if you use the Pebble framework and run on Tencent Cloud’s distributed computing platform, the complicated work is handled by Tencent cloud’s experienced team and automation programs. All you need to do is upload your application once and open the service. When business traffic is low, distributed computing platforms allocate fewer resources to run and save you money. When the number of business visits increases rapidly, the platform will automatically allocate sufficient resources to accept new requests. This expansion process is completely automatic. If your business falls back, the platform will also intelligently reduce running resources, reducing your overhead — this is the automated elastic scaling cloud capability.

Pebble framework Tencent Interactive Entertainment Business Group (IEG) R&D Department developed a general game server-side development framework based on years of experience in supporting Tencent games. The framework is very easy to learn to use, and can meet all the common game server-side basic functions with almost no dependencies; At the same time, it also has no need to modify the business logic code, you can set up a massive load of distributed cluster; Finally, this framework can cooperate with the distributed computing platform of Tencent Cloud to provide automatic operation and maintenance, rich operation and maintenance monitoring and log statistics, and intelligent elastic scaling of resources.

The open source address of this project: github.com/tencent/peb…

This article is from the official wechat account of HANu