Cfc4n · 2014/07/29 thy life
from:http://www.cnxct.com/experience-with-webgame-of-security-and-defense/
0 x01 background
During the holiday of October 1, I saw a question on Zhihu: what are the security problems of Web games? As a web game developer, I am very interested and impressed by this question. I was at play, and I didn’t have time to examine the problem. Later, a friend forwarded this question on weibo, and I felt an impulse to answer it. Last week 6, r&d department within weeks, hear the other an integer overflow problem, the project team lead to ShuaQian bugs, and reminded me of this problem, the more determined I want to answer the question of determination, to summarize the project, all the experience, the experienced webgame security to consolidate the current project safety barriers, to avoid the loss. Can also share to other friends who do Webgame research and development, do exchange and discussion.
The original question in Zhihu was “What security problems do web games have?” I changed it to “What are the security issues of web games and how can they be made more secure?” , at the same time, the problem also from “everyone to discuss, the full web game. Surely, this topic is very sensitive. At present, the web game has a lot of, will be made in black targeted? Web game will be black, the database will be towed library” changed to “explore the, everybody to study the full web game. Surely, this topic is very sensitive. At present, the web game has a lot of, will be made in black targeted? Web game will be invasion? Invasion way have? How to do web intrusion defense game? Saving measures have? How to minimize the manufacturer loss? Invasion way have? How to do web intrusion defense game? Saving measures have? How can we minimize vendor losses?” “, the reason for the change is that “the original questioner of this article mentioned at the beginning” let’s study and discuss the attack and defense technology of web games. “, it should not only mention how to invade, but also how to defend. It should carefully describe the formation principle of vulnerabilities and evasive methods to improve the skill level of developers; it should explain in detail how to minimize losses of manufacturers and users and protect game balance after security incidents occur.” Fortunately, zhihu passed this revision. Thank you for that.
In later reading this question and answer, there have been several netizens to answer, most of them are standing on the perspective of security workers to answer this question. In this log, I will take the webgame developer’s perspective, in line with the game business module logic, from business requirements, database design, programming, operation mode to explain the formation principle of vulnerability, evade the scheme, also welcome everyone to discuss.
0x02 Login Authentication
In recent years, web games are almost operated in intermodal mode, which means that the game server itself does not save user passwords. Users log in to the platform and log in through the interface between the platform and the game server. The interface performs encryption authentication. Therefore, the security of webgame account password is not mentioned here. However, the hash string security for login authentication is also important. For example, the effective time of the login hash string and the source of the encryption parameters of the hash string, such as the user name, login IP address, and user-agent of the browser, prevent the hash change from being leaked and are difficult to pass the server verification.
0x03 Game Recharge
Webgame game top-up process, with the average web top-up process is consistent, no special place, its differences with other platform for joint operations, each company is bound to do interface docking, and the interface specification of all kinds, and no voice game makers, they must be in accordance with the specification of the interface, it is really difficult. Tencent’s top-up interface authentication method, security is more prominent, about the code:
// Return a list of arguments $signKey = array('openid','appid','ts','payitem','token','billno','version','zoneid','providetype','amt','payamt_coins','pubacct_pa yamt_coins'); $sign = array(); Foreach ($signKey as $key) {if (isset($data[$key])) {$sign[$key] = $data[$key]; / / only GET some parameters, are involved in the calculation of sig}} # # # # # # to start generating signature # # # # # # # # # # # # 1: / / URL encoded URI $URL = rawurlencode ($URL); //2: ksort($sign); $arrQuery = array(); $arrQuery = array(); foreach ($sign as $key => $val ) { $arrQuery[] = $key . '=' . str_replace('-','%2D',$val); } $query_string = join('&', $arrQuery); $SRC = 'GET&'.$URL.'&'. Rawurlencode ($query_string); / / # # $key construction key = $this - > config - > get (' qq_appkey '). '&'; / / # # # to generate signature $sig = base64_encode (hash_hmac (" sha1, "$SRC, STRTR ($key, '_', '+ /'), true)); if ( $sig ! = $data['sig'] ) { $return['ret'] = 4; $return[' MSG '] = 'error :(sig) '; $this->output->set(json_encode($return)); return ; }Copy the codeOn this basis, we can also do rigorous points:
1. Add random parameter names and parameter values. Random parameter name and parameter value are randomly generated by the intercarrier and sorted according to the ASCII code of the parameter name string. Parameter name and parameter value are involved in the calculation of sign, which increases the difficulty of app key cracking by force.
2. Add callback verification order number and amount information. When the game recharge server receives the recharge request, it reverts to the callback interface of the platform to confirm the validity of the order, so as to prevent the problem of encryption key leakage.
0x04 Remote File Is Imported
In the development of web games, most of the framework is used to do, that is, to use the parameters from the REQUEST, as part of the REQUEST file name, so it is easy to form the vulnerability introduced by remote files. In our previous games, we had one such bug issue.
// Load the local application controller
// Note: The Router class automatically validates the controller path. If this include fails it
// means that the default controller in the Routes.php file is not resolving to something valid.
if ( ! file_exists(APPROOT.'controllers/'.load('Router')->getDirectory().load('Router')->getClass().EXT))
{
load('Errors')->show404('Unable to load your default controller. Please make sure the controller specified in your Routes.php file is valid.');
}
include(APPROOT.'controllers/'.load('Router')->getDirectory().load('Router')->getClass().EXT);
load('Benchmark')->mark('load_basic_class_time_end');
Copy the code
From the code and case figure, it can be seen that the parameters of REQUEST are not filtered, and are directly included as file names, thus causing this problem. Similar to the above page figure, if PHP version < 5.3.4, Null(%00) truncation will occur, bringing greater security problems. In our new project, we have changed the implementation method. All interfaces in our game will go through gateway. In gateway, we will do the detection processing of the controller name and the class name specification, and then do the Autoload loading file in several designated directories. In addition, it uses ReflectionClass to reflect the class name and method of the REQUEST, and checks whether the class, method, and parameter are valid. One is to avoid the “remote file introduction” vulnerability problem, and the other is to facilitate the front and back end debugging, throw more detailed exceptions, convenient debugging. The following is the reference code:
require_once CONFIG_PATH . "/auto.php"; spl_autoload_register("__autoload"); ... $view->clear(); $view->error(MLanguages::COM__INVALID_REQUST); $msg = new Afx_Amf_plugins_AcknowledgeMessage($val->data[0]->$messageIdField); $msg->setBody($view->get()); $message->data = $msg; ... $a = new Yaf_Request_Simple(); $a->setControllerName($method[0]); $a->setActionName($method[1]); $objC = new ReflectionClass($method[0]."Controller"); $arrParamenter = $objC->getMethod($method[1]."Action")->getParameters(); $arrRequest = isset($val->data[0]->body[0]) ? (array)$val->data[0]->body[0] : array(); $bCanCall = true; foreach ($arrParamenter as $objParam) { $parm = $objParam->getName(); $bIsOption = $objParam->isOptional(); If (isset($arrRequest[$parm])) {$a->setParam($parm, $arrRequest[$parm]); } elseIf ($objParam->isOptional()) {else {$bCanCall = false; } } if ($bCanCall) { $rp = $app->getDispatcher()->dispatch($a); $msg = new Afx_Amf_plugins_AcknowledgeMessage($val->data[0]->$messageIdField); $msg->setBody($view->get()); $message->data = $msg; }Copy the code0 x05 SQL injection
The principles and methods of SQL injection are the same as those of common Web applications. When using parameters from REQUEST, you can filter them. The format of the message, and the ease of injection, may have blinded the developer and been overlooked. For example, in the AMF message format of our project, before the front-end interface came out, we back-end programmers generally used Pinta to simulate operations and debug programs. After the front-end interface comes out, the Charles Proxy is used to capture HTTP requests. In these processes, the request interface, the construction of parameters, is not as simple as the normal Web. It is also easy for developers to ignore filtering request parameters, so this problem can easily develop. See the formation principle: WEB Development security and Operation and Maintenance Security Overview, defense mode to do filtering, or SQL precompilation.
In order to improve the capacity of the game server, the web game architecture has been evolving. In the previous Webgame architecture with Mysql as the data store, other nodes can be expanded horizontally, or rely on simple and crude addition of servers to solve the problem, which cannot be done only as the sole data storage center. For this reason, a lot of Webgame data storage to Nosql to replace, and even Java, C/C++ game data, directly in the memory operation, when the game is closed, written to the DB. Therefore, SQL injection problems will be less and less.
0X06 Communication Protocol and Message Format
Although web games are called web games, the communication protocol is not all HTTP, there are also many uses of socket, and HTTP +socket. We are HTTP protocol + AMF message format, and socket and used to implement. In terms of the choice between HTTP and HTTPS, we consider that after SSL is enabled, a large number of SSL decryption and encryption operations will inevitably increase the CPU calculation pressure of the server. The content transmitted, mostly the operations and responses of the game business, is the behavior that can be sniffed by the listener (except for authentication information). From a security standpoint, it doesn’t make sense. But from a product perspective, it makes sense to consider the input-output and choose HTTP communications. Socket in our game, in addition to chat applications, we also use sockets to push data when multiple players need to synchronize data information between platoons and gang battles. When socket is used as the protocol for all service transmission, the protocol format is generally open source protocol, such as MSgpack, Protobuf, or custom protocol. When using the custom protocol, be sure to detect each parameter and type range of the whole message package to avoid the occurrence of a few large values and boundary values, which will lead to the memory of the main program exceeding the boundary, so that the service will break down and the normal service cannot occur.
0x07 Gold Copy – Integer overflow
In the weekly meeting held on Saturday last week, I heard about the problem of flushing gold caused by integer overflow from other project teams. Here, I abstract the case and share it with you. The mall sells “sycamore wood,” the item needed to open the backpack grid. In a game, the number of squares a user wraps up is usually used as a charge point, and the number of squares in a game is about 7 squares per row, for a total of 8 rows. For example, the front 3 lines are open by default, and the fourth line is charged, and the price of the first grid product wutong wood is 1 silver, the second wutong wood is 2 silver, and the third is 4 silver. By analogy, it means that the sum of the prices of these wutong trees is actually a geometric sequence with the first term 1, the common ratio 2, and the term 35. When the user chooses to buy more than 31 sycamore trees, such as the numbers in 32-36, the sum of these geometric sequences is greater than 2147483647. (Just for example, items don’t actually sell at these prices)
In Java, 4-byte ints can be stored in the range -2147483648 to 2147483647. When stored in Java, C signed int type, the highest bit of the number describes the symbol bit, 4 bytes in total 32 bits, remove the highest bit of the symbol bit, the remaining 31 bits, each bit can represent two digits, 4 bytes of signed integer representation range is: negative integer 2^31, the range is “-1 to -2147483648”; The value is a positive integer 2^31. The value ranges from 2147483647 to 1. Here’s an example (note the order in which decimal and binary representations change) :
When the number of open squares is greater than 31, such as 32, the cost will be 2,147,483,647 silver taels. Then buy other items to make up the number more than 2,147,483,647 silver taels. For example, buy 3 other items of silver, and the total cost will be 2,147,483,650 silver taels, which is represented in a 4-byte signed int. The sign bit becomes 1, which is a negative integer. The value bit is 0000000 00000000 00000000 00000010, that is 10000000 00000000 00000000 00000010, corresponding to -2147483646 in decimal notation. Program logic, and then determine whether the existing silver is enough to cover the cost, yes. When subtracting the cost from the current balance, it becomes subtracting a negative number, which is essentially adding a positive integer. It becomes an increase in the balance of your bank account. While the balance field type is long, these balances are correctly stored and the overflow vulnerability is exploited. In C, you can use unsigned numeric types to do overflow and swipe, but in Java, there seems to be no unsigned types. It can also be determined that all values involved in the calculation must be positive integers as a necessary condition (game business features, all numbers in the game must be positive integers, even excluding zero), and then do the size judgment, plus plus, can not get negative; You don’t get a plus when you add the negatives. To determine if overflow problems have occurred. In PHP, you don’t have to worry about overflows. For more information: Overflow, Symbols, and Carry
If you use an unsigned int in a database, set it to unsigned int/tinyint/… if you use an unsigned int, set it to unsigned int/tinyint/… . In the game, numbers are almost all natural numbers, and negative integers are rarely seen.
0x08 Prop Copy – Logic error
In my new Webgame, there is another problem of brushing props. The problem occurs in the business of moving props in the backpack. The business needs are as follows:
When moving from position A to position B, if both are of the same type and can be stacked (both are bound or unbound), the amount is stacked to one and the other is destroyed. If they are not of the same type, or cannot be superimposed, the two call for position. If the target position is empty, move the item from A to B. How would you write this function if it were written by you? All the code in the gist: programmers b7d794d3cb663dd86aad may be long, actually this problem in the code below
@param string $former * @param int $former */ public function MoveItemAction($former, $target) { $former = trim($former); $target = intval($target); if(! $former || $target < 1) { return $this->_view->error(MLanguages::COM__PARAMETERS_NOT_LEGAL); // The parameter is abnormal; $former_item = $this->__mItems->getNumberItem($former); if(! $former_item) { return $this->_view->error(MLanguages::PACK__NOT_EXIST); // Use item does not exist; } $former_md5id = $former_item['md5_id']; $former_num = $former_item['number']; $former_base = $former_item['item_id']; $former_sup = $former_item['superpose']; $this->__mRole->beginTrans(); $target_item = $this->__mItems->getNumberItem($target); .Copy the codePart is the problems such as these, when they look at the code, will think that this preliminary judgment is right, in conformity with the business logic, the problem has happened “source grid” and “target” grid judgment, not whether these two parameters is the same location, if the same position, so the above all business logic is legitimate, the following code, The “two” items will be stacked according to their properties, and then the item duplication problem occurs…..
The fix, I think you all know, is to determine whether the source location and the target location are the same
if(! $their weight.this | | $target < 1 | | ($their weight.this = = $target)) / / to determine whether a source of target location is the sameCopy the codeWould you make such a mistake? I want to… Ha ha…
0x09 Gold Copy – Concurrent request
Most RPG-style web games have the ability to sell items, sell them directly to the store, and buy items and materials from the store. When a large number of concurrent requests are made by players for both buying and selling operations at the same time, the processing logic of the server is generally handled by two separate processes, and the corresponding account balance table in the shared data database is shown as follows:
$iBalance = $obj->getBalance('user1'); UPDATE 'role_gold' SET gold = 150 WHERE role_id = 1 if(! $obj->setBalance('user1',$iBalance + 100)) {//rollback} $obj->delItems($items)) {//rollback} //commit // buy // startTrans $iBalance = $obj->getBalance('user1'); UPDATE 'role_gold' SET gold = 0 WHERE role_id = 1 if(! $obj->setBalance('user1',$ibalance-50)) {rollback} $obj->addItems($items)) { //rollback } //commitCopy the codeThe sell request is processed by process 1 and the buy request is processed by process 2. Process 2 also reads a balance of 50 from DB before process 1 writes the result to DB. This is, the balance information for both processes is 50. According to the logical code, process 1 calculates that the remaining balance is 150; Process 2 calculates that the remaining balance is 0. Finally, whatever process ends up writing the balance is the wrong result. (Note: this code does not have anything to do with mysql transactions. Transactions can only ensure that multiple statements within the transaction scope of a single process are executed correctly, or rolled back. For example, two statements that ensure that the money is deducted and the item is deleted are executed correctly. Ensures that all statements fail and are rolled back correctly. Whether the SELECT statement can fetch the latest data or wait for the lock to be released depends on MYSQL’s transaction isolation level. There are several isolation levels in MYSQL transaction isolation:
Read-uncommitted Indicates the level of reading UNCOMMITTED content
Read-committed (READ the COMMITTED content
REPEATABLE-READ
SERIERLIZED(serializable)
For read-uncommitted, data can be READ from other transactions that are not committed, and the performance is said to be no better than that, so it is rarely used in real life. For read-committed, the same SELECT can return different results in the same transaction because another transaction may have a commit at any time. This doesn’t work in the game business either; In SERIERLIZED, once the transaction is started, all other queries are queued up for the transaction to commit. In the case of the above mentioned buy and sell case, the SELECT operation of the second transaction will not return immediately and will remain locked until the end of the previous transaction. This isolation level, while avoiding the above problems, has poor performance and is generally not used. REPEATABLE-READ isolation level is also the default isolation level of mysql, which is more in line with the needs of game business from the function, and should also be the default isolation level of mysql in the vast Webgame architecture. When REPEATABLE-READ isolation level is used, the select data is the data before the transaction is submitted, and each transaction can be successfully executed normally, so the error result is executed.
You may have a quick solution to this problem, UPDATErole_goldSET gold = gold + 100 WHERE role_id = 1 or UPDATErole_goldSET gold = 150 WHERE role_id = 1 AND Gold = 100, but multiple transactions that modify multiple records of multiple tables at the same time can also cause Deadlock problems, as shown in “Troubleshooting for Mysql Deadlock ERROR 1213 (40001) in Webgame”. Also, what if the condition is whether data exists across tables, or if another condition is not in MYSQL but in the response of another network interface?
0x10 Gold Replication — Logic Vulnerability
Citing DNF’s vulnerability news, “The player who made money by using online game vulnerabilities revealed that he earned 170,000 yuan in 3 days”
A game item can be used for 400 RMB. What is this bug? The original game “cloud power pocket pot” this props, can open 2 pieces of the same game equipment, and very little chance to open game currency, out of the equipment is not valuable, but if the gold coin, is divided into 50 million, 80 million and 100 million game currency. And 100 million game coins, according to the normal market prices, can be sold on the trading online 400 yuan. According to players, in the game, a character’s equipment needs to be stored in a package, but currently the character’s package is limited to 48 compartments, meaning it can only hold a maximum of 48 items of equipment. The vulnerability is to use the limited space of the package to store 47 pieces of equipment (when the package is full, the jar cannot be opened), leaving only one empty space. When opening the “Cloud power pocket jar” to open the equipment, it will fail due to the insufficient space of the package, but the jar still exists. The player continues to open the can until coins appear, but the coins do not take up space in the package, so the can is successfully opened and the can disappears. After the bug was discovered, some players spent the game coins and then immediately sold them on third-party trading platforms for cash.
This kind of problem is caused by unreasonable logic of r&d personnel, and this kind of problem is also difficult to find. Circumvention can rely on “operational data monitoring” as mentioned below.
0x11 Item Duplication — Pack sorting
Do the same with selling and buying, while wearing equipment and packing. In the design, the body equipment may be designed in the equipment list; Design items that are not on your body into your backpack list. When to wear at the same time equipped with tidy package requests concurrently, can also occur with the sell, buy, thread 1 DB, found in the parcel with this equipment, and then to delete this backpack tables records, when preparing written to the equipment list, another tidy package request thread here, read the entire bag table, merging, sorting, props. At this point, the previous thread writes the device to the equipment table, deletes the data in the backpack table, and commits the transaction. All operations of this outfit are reasonable, normal, and performed correctly. But another thread of packing read the data from the previous packing list, including the item being worn. In the game, packing requires stacking stackable items, which means merging multiple items and removing some items. This means that the operations here, the data in the memory of the current CGI thread, will be written to the DB as an overwrite, which means that the piece of equipment that was previously worn will be written back to the backpack. That’s an item with the same property in two tables with the same unique ID. The player can then sell the item in his backpack to other players.
In Java or C programs, games where data is stored in memory will also have this problem unless a read lock is done, but the read lock will cause the lock wait, which will cause the thread to be occupied, blocking the processing of subsequent requests, and piling up a large number of requests. As a result, the system load increases and the server becomes too busy to respond. Ok, do you understand how item duplication works? This problem, we think from the root cause, where is the problem? How do you get around it? It is not difficult for careful students to find that the operation result of wearing equipment will affect the next request. The server cannot process the next response before the current operation gets the response from the server. For this, we do a response processing lock – “user concurrent request lock”.
When a user requests a lock concurrently, PHP locks the session file and does not release it (unless session_write_close is executed) until the current response is complete. This creates a lock for concurrent requests from a single user. However, subsequent processes remain in a waiting state, blocking the use of the PHP thread for normal requests from other users. For this purpose, we use the K-V structure of NOSQL, with the form of user_name as the key, to implement the user to request the lock concurrently, such as redis setnx interface, atomic judgment operation if there is false, not add a, return true. So, for the next request, setnx, return false, have this key, so immediately throw an exception, end the response, FLASH based on the exception content, to remind the user not to do malicious operations. Neither concurrent requests occur nor request processing is blocked. At the same time, in the destructor at the end of the request, the lock is deleted without affecting the next normal request. If the destructor cannot be executed because of a syntax error caused by an abnormal program, if the user lock is not deleted, you can set the expiration time when the lock is generated, such as 5 seconds, or even 2 seconds. The expiration mechanism of NOSQL can be used to unlock the user and prevent the user from being unable to play normally for a long time.
0x12 Class CC Attack – TimeBomb for Multi-User shared resource locks
The project we are developing now uses NOSQL Redis as DB to store data. Redis has no mature transaction processing mechanism, and Watch is not even transaction processing in relational database. In this case, it is necessary to lock the table to unlock it. Java and other language projects, many are directly operating memory, but also require resource locks, to solve the concurrency problem, to solve the problem of multiple requests to operate on the same data. In another project of the company, there was a timebomb problem of lock waiting due to the large granularity of the lock, which also led to the problem of busy threads, congested, requests piling up, system load rising and downtime. The lock for this project is for all users. When each user’s request comes in, the current thread locks all users’ data until the response is complete. This is done to solve the problem of affecting other user data due to the current operation, such as multiplayer PK, interaction between multiple players.
As soon as other requests come in, the resource is locked until the last request ends and the lock is released, leaving all other threads in a waiting state. When the user base is small, you can’t see the impact of locking. Memory operations are faster. When the user base is large, or the number of requests increases, the waiting time of subsequent requests will become longer and longer. When the waiting time of webserver exceeds, timeout is returned directly, and services cannot be provided normally.
This problem occurs because the granularity of the lock is too large to lock all users. It is best to break down to a single user affected by the current request and only lock data for a single user. In this way, timebombs are reduced.
0 x13 other
My friends in Zhihu mentioned that the front end of a lot of Webgame makes a judgment, but the back end does not make a judgment. This kind of problem should not exist. In our case, the back end did a lot more validation than the front end. Sometimes, in order to animate the interface, the front-end Flash will render immediately after the user does something, and then, depending on the back-end response, decide whether to continue to make interface element changes. For example, when unarming, the player will render an animation of the equipment jumping from the character palette to the backpack, and then decide whether to scroll back based on the backend response. In this way, avoid appearing to be insensitive after a certain time of operation to improve the user experience. Of course, the back end will definitely make a judgment about whether the character backpack has Spaces or not. Now webgame research and development, there is generally no front-end judgment, the back-end does not judge the practice. If so, it should be an isolated omission.
For example, last year’s Time33 algorithm hash DOS issue, webgames that use JSON message format must be noted that PHP only accepts a maximum number of requests. But in json decoded data, there is no processing. Don’t forget that here.
0x14 Abnormal Operation Data Monitoring
Even the best defenses will still have security holes. Appropriate monitoring measures must also be taken, such as monitoring levels, gold coins, game coins, experience, changes in precious items, etc. Once found, immediately alarm, before the vulnerability has not spread, the first time to repair the vulnerability, in order to reduce losses and maintain the game balance.
0x15 Log System
Log system must not be missed, all operations, must be written into the log, when the security event occurs, can be used as a variety of data rollback, transaction dispute processing reliable data. It is also the most accurate data source for data monitoring.
A true story. In June last year, our project launched a new system and upgraded Tencent recharge interface V2 to V3, which involved the change of recharge code. After r & D test, planning test and QA test, we launched the system to individual service areas and observed the situation. Every time a new version is launched, the whole project team will continue to observe the data situation, especially the total recharge, 10, 50, 100 rise, suddenly, the total decreased. Recharge total declined, but this is the total ah, can only increase, will not drop. UPDATE table set gold_num = $num,is_pay =1 WHERE gold_num = $num,is_pay =1 What a stupid BUG, can you tolerate it? I’m gonna pull it out and shoot my dick to death. When I saw the modifier involved in this file in the SVN log, I immediately petrified, quietly fixed the bug and uploaded……
It did happen, and the reason was stupid. I forgot to write the WHERE condition, and because the framework is wrapped, the problem is not easy to see. In addition, the recharge of my own test was correct, including the later planning test and QA test, but no problems were found. The credit goes to the “data monitoring system,” so we, the game developers, found out about the problem before the users did. Data monitoring must be essential.
As for fixes, thank the logging system. For each recharge, there are double logs, one is in the DB of each game area. The second is the recharge center. The recharge center is responsible for docking with other major platforms. This accident, affecting the game area of the data, did not affect the recharge center data, so it can be checked. This way, great DBAs can repair data more easily and safely. To this, regardless of the occurrence of other money, brush equipment, theft after the transaction dispute, can rely on the log to do processing. A log system must also be essential.
It took less than one hour to release the new function, discover the BUG, and fix the BUG (the SVN time can be seen, 16:30 external). You can imagine how commendable the effect of data monitoring is. When a new feature goes live, each testing session is done. According to the large area server, do grayscale release can also be smaller to reduce the scope of bug impact, reduce losses.
If you use any of my small, refreshing illustrations, please be sure to put your signature on the picture. Drawing is one of the most time-consuming things I do. By the way, if you see the reprint, I suggest you come over to see the original text. On the one hand, the typesetting format of the original text will be more suitable for reading. On the other hand, the original text will continue to correct the errors in the text and add content.