On May 7, “Tencent SNG & MSUP Technology Open Day” was held in Shenzhen. Yibai interviewed some of the lecturers to talk about what they will share at the conference. The lecturer we interviewed in this issue is Fu Hongcheng, an expert engineer from Tencent and the technical director of QQ Music.
Yibai Case: Please briefly introduce your current work, the field you are interested in, and your technical accumulation.
Fu Hongcheng: The personal growth process and the technology field that each stage focuses on can be seen in the following figure.
I have some experience in the field of Internet technology, and I am good at streaming media operation and optimization. From Web architecture to client architecture, from traditional Internet to mobile Internet, I am interested in learning and mastering new fields quickly. MainTAINS attention to business operation data, continuously optimizes architecture and improves service quality through three-dimensional monitoring, has a technical vision for public component planning and general architecture design, and considers long-term product planning in technical implementation.
Yibai case: what you do is closely related to the architecture, can you talk about your understanding of the architecture? And how do you define an architect?
Fu Hongcheng: I have been working for eight years after graduation. I have been responsible for a lot of structure design and system optimization. I have some experience summary. There are no standards for architectural design, and patterns are constantly changing. The absence of standards in architectural design is not that there are no standards, but there are too many. Architects all want to find the best design solution, but there are different understandings of what is best.
A good architect, before designing a new system or refactoring and optimizing an existing one, needs to prioritize the requirements of the system. We can clarify through a series of questions, better see the host, such as: this demand must be done? The system cost that brings has how old, have without flexible method? Can we do it next time? What do you lose by not doing it? If you redo the old system, many of the old requirements may not exist, which can reduce a lot of constraints, get rid of these historical baggage, and the new design can be lightweight.
In addition, as an architect, it is necessary to consider whether there are similar systems in the company and the industry before designing the system, which is also the basic ability requirement of Tencent for senior engineers. In particular, it is necessary to avoid repeated construction of cars. It is better for you to stand on the shoulders of giants as soon as your system comes out.
The architect should also keep learning about the internal and external architecture, including excellent technical forums and summits, so that inspiration will flow when designing a new system.
Internet architecture design follows some principles, big system small work. Simple is designers through the deep understanding of requirements and design, so as to keep the system simple, simple is not equal to humble, such a system is relatively controllable ecological, can be a high degree of understanding and good ability of extension evolution, if it is a complex system will also be necessary to split into concise subsystem, set a clear and simple interaction between subsystems.
I am also a judge of the company’s R&D promotion management committee. Our requirement for senior engineers is to have everything under control, that is, to conduct in-depth analysis of all aspects in the design stage, make correct decisions, and ensure that the final results meet the design objectives.
Architects, for example, that the background, you need the background service system has a deep understanding of the entire operation process, network request packets received from the network card, the response to the final network packets from the network card to send out, what happened in the whole process, the various components of a computer is how to collaborate on, where is the bottleneck, expect it can achieve what is the load capacity, etc.
In addition, estimation is one of the fundamental skills of an architect. Estimation can not avoid all problems, but it can avoid making bigger mistakes. For example, you need to have a good understanding of memory sequential read performance, SSD read and write performance, MEMSET1K, SHMAT data time, etc., which is the foundation of predictive ability.
Most programmers who have just started work will have the heart to become an architect. They will keep the habit of accumulating and summarizing and ask for advice from old colleagues with a learning attitude. In the next few years, they may have the opportunity to independently take charge of a system and become an architect.
Yibai Case: As the head of QQ Music R&D, how do you arrange your time for new technology learning, R&D team management, programming, life and so on?
Fu Hongcheng: to write code to tell the truth now is not much, comparison of guilt, hope have a chance to still write more, remember that the first three years after graduation system statistics to myself in a new or modified qq music more than 60% of the back-end code, code of new and modified quantity or top ranking in a couple of years later, but the team are also a lot less than now. Right now I’m still involved in the code hierarchy in Code Review and Bug Fix.
As for the R&D team management you mentioned, here is a brief share of what I think a technical leader of a team should do and how to allocate his time reasonably. As a team leader must first have a clear understanding, with the main transactional work out and need to give up some appropriate management of power ability, in order to improve the team overall development quality and efficiency as the main target to arrange their own work, and at the same time does not affect the subordinate core team leader for the development of space especially, on some things need to identify on power or not. Generally speaking, the technical director is responsible for the main process and control of some technical projects.
The work of the main program includes development guidance, personnel training, and management. The development guidance work includes proposing non-functional requirements, designing and revising the software architecture. We have also seen that many Internet start-ups are stuck in research and development problems due to non-functional requirements such as performance, product quality, scalability and secondary development efficiency that have not been seriously addressed. As one of the most experienced members of the team, you must use your experience, especially lessons learned, to come up with those non-functional requirements to keep the whole project from collapsing after release.
In addition, personnel training is the core means to stabilize team members, and this part of the work should take up about 20-30% of the time, including code review, architecture review, and technical lectures.
The R&D teams I am in charge of maintain biweekly technology sharing meetings and monthly technical exchange meetings of the development center, which are relatively good among all the teams in SNG. I think it has a good effect on the stability of the team and the growth of members.
In terms of team management, I pay attention to goal orientation and process, and take improving the overall efficiency and quality of the team as the core goal. Technical team management need to innovate, come up with new ways to solve the problem, usually in some team members to implement development daily behavior including research and development of quality indicators, take the initiative to share, take the couple etc. To evaluate the performance appraisal for every half a year’s team members to provide certain reference, moreover as head technology across different teams and departments of technical cooperation and advance is also an aspect.
After you arrange your work properly and improve your efficiency, you will have time to spend with your family. Now I will leave some time to spend with my family. Recently, I also have a lot of time to play group games with my friends. But frankly speaking, even when you are with your family, you need to pay attention to your business all the time, so as to avoid emergency response to unexpected business failures and temporary problems. This is also a basic embodiment of responsibility and dedication, so are most employees of Tencent.
One hundred case: how many time nodes or nodal events in the growth and development process of the QQ music business architecture? Is QQ music architecture in the growth stage or mature stable stage? And what are the technical challenges?
Fu Hongcheng: Briefly talk about the last three or four years, the last four years is also the most rapid mobile Internet changes a few years.
At the end of 2011 and the beginning of 2011, QQ Music also rapidly expanded from the original PC QQ Music to mobile phone QQ Music and each terminal, and became the main battlefield;
In 2013, we made a special optimization of streaming media, which completely solved the problem of slow download of listening to music.
In 2014, we focused on mobile terminal experience optimization, and achieved breakthrough improvement in the performance of music library release system. The release time of information input, transcoding, distribution and full-platform effective was shortened from one day to half an hour and even now to quasi-real-time effective, which laid a good technical guarantee for QQ Music’s operation of a large number of TV programs.
In 2015, a data center was established to sort out the data of thousands of copyrighted music libraries, and to refine the basic data, quality of music libraries, classification stations and other aspects, thus establishing a good reputation among users.
The QQ music business is still in a period of rapid growth, and I think the same is true of the technology architecture, which continues to grow. Future will face a lot of the technical difficulties, intelligent recommendation, continues to data mining field, for example, we recently in machine learning and training in the lyrics, to extract the lyrics central idea further training similar to single group, etc., from audio spectrum similarity and singer before make up for the overall style of the insufficiency of the classification, foresee a good effect.
At present, Docker is already being used in the transcoding service of QQ Music. In addition, we are also continuously paying attention to the excellent technologies of excellent Internet companies at home and abroad, keeping the attitude of learning to understand and contact with them as much as possible, and we will also keep paying attention to and learning from those that are available to good technical teams.
Yibai Case: How does SNG work with experience as the center? Compared with other businesses, what are the particularities of music, such as streaming media business, in terms of technology breakthrough?
Fu Hongcheng: Very good problem, here I want to say more. Many optimizations of SNG are not at the cost of user experience loss, including equipment and bandwidth optimization, which is also realized by optimizing the system architecture and improving the rationality of the architecture under the condition that the experience is not affected or even significantly improved.
In the era of user experience as the center of the Internet, any development activities should be to improve user experience as the ultimate goal, of course, technical performance optimization is no exception. Optimizing product performance most avoid falling into the pure pursuit of technical limits and optimization situation. QQ music streaming media structure we have also had a special attack, to experience as the center of technical optimization deep feelings, can be described in detail.
I still remember that before we set up a joint team to do special optimization more than two years ago, although the overall music listening was relatively smooth, there was a lot of gap compared with foreign industry competitors (such as Spotify), and some performance indicators even lagged behind some domestic competitors. CDN relies on one CDN, which leads to uncontrollable performance, efficiency and risk. As a result, there are often a group of users who continuously feedback the music card, unable to listen to the music and slow download speed.
At that time, we also realized that QQ Music should maintain the first place in the industry and set a high threshold for the industry. The performance and quality of listening music is an important link. In the face of China’s complex network environment, how to enable users to listen to songs in seconds and fluently is an optimization subject that needs continuous investment.
Finally, after we put a lot of energy into stream-media performance optimization, the core performance indicators were significantly ahead of competitors, and users gave very positive feedback. At that time, the joint team also won the Tencent company level excellent technology operation bonus award due to the remarkable optimization results. This goes hand in hand with all the technological optimizations that are experience-centric.
First of all, the goals of the joint team were formulated with the core members of the team before optimization, and these goals were successfully achieved through the efforts of the team of client, background, P2P, basic data and other modules. The goal is to focus on the user experience, so that QQ Music users across the country can listen to songs at a moment’s time, so as to achieve the first smooth experience in the industry. In this process, a unified quality control and resource scheduling management platform is built to improve operational efficiency and reduce operational risks.
Developers use after about 3 weeks, dedicated to collect user about listening to slow on the BBS, download card’s feedback, and one by one to contact the user, go deep into the colleges and universities, second-tier cities network research, in order to optimize the gained valuable direction, our team is a newly hired technical backbone zack have contact for a period of time focus on the user, Including applying to join some QQ groups of colleges and universities, and constantly asking them what problems they encountered when using QQ music to listen to songs, and inviting students to grab bags if they encountered a delay, so as to understand the network environment of students.
We set out to address the pain points of the experience that plagued users: the need to wait until the start of online listening; Listening to the song in the process of the card; Listening to a song has an error; Slow download speed; The song doesn’t download.
After that, the team prioritized the experience indicators, assigned different weights to each indicator, and multiplied the weight value by the degree of completion of the indicator to get a score for each experience. Each experience score is then added up to get the total experience score. If the experience score increases, we can assume that the user experience is improved; Otherwise, the user experience becomes worse.
Taking this optimization as an example, we extracted four experience indicators, including the probability of lag in the process of listening to music, the buffer waiting time before listening to music, the speed of downloading songs, and the error rate of downloading songs, and then sorted them by priority and assigned the weight values of 40%, 25%, 20% and 15% in turn. The priority of metrics can be adjusted in different stages of the product according to the product strategy. For example, in order to save traffic, mobile users are more likely to download songs to local sites and listen to them later than PC clients. This makes download speed an even more important experience metric for mobile users than it is for PCs.
In order to monitor the actual optimization effect generated by different optimization strategies, we have built a set of massive user data analysis platform, which is used to monitor, calculate, process and display billions of users’ listening and downloading data every day on the server side. The macro listening data of all users is calculated into the average value of core experience indicators, which is further calculated into the core experience score.
By monitoring the trend of the macro experience score, you can control the effects of different optimization strategies. After the core experience indicators are established and the experience score monitoring is completed, the optimization strategy can be proposed and implemented. Finally completed the 30 a number of large and small optimization point, including the access layer as far as possible by racing access, user data cached in the nearest place as far as possible, try to predict the behavior of users and to get the data in advance, as far as possible according to user’s environment to provide customized parameters such as the international and domestic innovative experience in streaming media optimization.
In this process, we also applied for nearly 20 streaming media technology patents. Due to the space, the detailed technical details will not be carried out. Here, we also want to thank the streaming media group of Tencent Architecture Platform Department for their strong cooperation and support. Returning to technical optimization itself, optimization always focuses on improving user experience, otherwise it is easy to fall into the dilemma of making insignificant optimization in order to show the technical strength, which wastes resources and damages morale.
Yibai case: I took the position of technical team manager. Could you talk about your research and development team size, software development method and management experience in different stages?
Fu Hongcheng: I have already answered some of the questions in front of this point, so I won’t talk more about it here. At present, I am mainly in charge of a team of about 100 people, and I have different management ideas and methods at different stages.
Team size large, not necessarily every member you can be a very good attention to what they think, what are the requirements, the greatest degree to keep the stability of the team core backbone for the development of the business is good, constantly motivate them, through party building, the training of organizational learning, to share, that make them feel up and also my team three-bottle man many, Invincible.
This article from the “one hundred cases” public number, link to the original