Summary: Cloud native is redefining storage, and storage is just one piece of the underlying software landscape. In this c-bit face to face, we have the honor to invite Jason Wu, senior researcher of Alibaba and head of Intelligent storage of Ali Cloud. He has witnessed the rapid development of Ali Cloud storage technology for 8 years. This time, he will explain the present and future of storage technology with InfoQ. As well as his own career in the face of some choices and personal growth.

Deeply cultivate the foundation for twenty years

If you look at the history of IT technology, each period has a different focus: 30 years back, high-performance computing, 20 years back, search, and 10 years back, cloud computing. Having worked in the Institute of Computing of Chinese Academy of Sciences as a supercomputer, ASK.com as a search engine and Ali Cloud as a cloud storage engineer, Jason considers himself a lucky person in his career choice — he has never been absent from the development of the three key technologies of high-performance computing, search and cloud computing. “In the future, AI and machine learning may be another new technology development point.” Now Jason has become the second party of AI and machine learning. The storage products he is responsible for are supporting new loads such as big data analysis, AI, machine learning and autonomous driving.

Jason joined the Microsoft Azure team in 2008 and was involved in writing Azure code, including subsequent operations and on-line troubleshooting. “Since I wasn’t part of the original wave, I was part of the Azure process from 0.2 to 1. The original wave was on the team in 2007. I think I was very fortunate over the years to be involved in the evolution of such a large product and technology system at an early stage.”

“I remember in 2012, WHEN I came to AliYun for an interview, Academician Wang Jian asked me,” You have worked in Azure for 4 years, what do you think is your greatest achievement? Without thinking about it, I said my biggest accomplishment was making the system a little more stable. I used to have to work seven days a week, take my computer with me every time I went out, and sometimes I had to find a Starbucks on the way down the road. From 2008 to 2011, there were a lot of such cases, but by 2012, our system was basically stable, so there were fewer cases. It could be six or five days (a work week), and there were fewer times when you had to take your computer with you to check problems.”

“Later it was reported that the Doctor was satisfied with my answer and said that there were indeed a lot of problems to be solved and holes to be filled in the early stages of the system.”

However, Jason did not choose to join Aliyun immediately after receiving the Offer. “Because when I gave the Offer, there were two choices: one was to go back to Hangzhou, the other was to let me go back to Silicon Valley. Ali Cloud has a RESEARCH and development team in Silicon Valley and wanted me to lead that team.” But he prefers to stay in Seattle. Therefore, it was not until ali set up Office in Seattle in 2014 that Jason took over the olive branch after two years of hesitation. However, it was because he saw the prospect of cloud computing in China that he finally made the decision. Compared with chips and operating systems, cloud computing is a technology that China can do well.

“But today my understanding has changed a little bit. In fact, we can also do something about chips, like operating system. Ali Cloud is also doing its own operating system. As we get deeper into chips and operating systems, it can actually do a better job of getting to the full stack.”

Experience at Aliyun: Three major decisions and lessons

From 2014 to 2022, Jason has spent 8 years with Ali Cloud as the head of Ali Cloud Storage. Along the way, he has been involved in many important decisions and learned many lessons. Here are three stories he will share with us.

  • The biggest decision of 2015 was to send the best people to write the SDK

The Flying Sky system was impressive, but in 2015, the business development of Alibaba Cloud object storage was not satisfactory. “We all thought our distributed system was great, our Flying operating system was great, our Pangu was great, our products were great, and then we reasoned that because we had great products, our customers would love to use them.”

But that’s not the case, so what’s the reason?

“We found out that our backend was great, but we didn’t do the right things with the customer, the SDK wasn’t very good, and the user experience wasn’t very good.”

“For example, you can spend a month making an optimization that improves the performance of the entire system by 50 percent, but because of a piece of code in the SDK that isn’t as well written, the customer may not get that 50 percent [optimization]. We, the elite programmers on the back end who do distributed systems, don’t get that.”

So, the road closest to the customer needs to be very good, which is the biggest lesson that The Alibaba cloud storage team learned in 2015. “It’s like someone has a mansion, but if the roads to get in are full of thorns, there’s less access for everyone, so you have to build those roads, right? Things like our SDK, our console, these things are actually those paths.”

With that in mind, the Alibaba Cloud storage team made one of the biggest decisions of 2015 at the end of the year — sending the best people to write the SDK, focusing on console builds and response times, as well as NPS, which is the willingness of customers to recommend their purchases to other friends. “We put our best students in that area and made the SDK and console really user-friendly and high-performance, and then we got better at those details and our business took off.”

If you are also a developer and a back end student, this story should inspire you. From the perspective of the whole link, only by doing every detail well can we ensure that the advantages of the backend can be truly enjoyed by customers.

  • In March 2019, the darkest hour of Ali Cloud storage team

In March 2019, the darkest moment inside Ali Cloud storage team. Within a month, Aliyun suffered two major failures. One was a major failure of block storage EBCS in the availability area C of North China 2 region, which resulted in service interruption for several hours and multiple apps and websites crashed. A week later, ali Cloud object storage OSS also suffered a level P1 failure.

At that moment, Jason truly realized what is called “the foundation is not strong, the earth shakes”, once the problem of the infrastructure base, it will bring a huge impact on the upper customers. “We need to make the system more stable, take our quality of service, take our quick response, take our failure recovery to the next level. “Customers have invested their lives in Aliyun, and it is our responsibility to provide them with a system that never breaks down through technology and services.”

Even three years later, the whole Ali cloud storage team remembers this month very well. They would even repeat these cases to new students, telling them to Revere every line of code, every action.

  • In 2020, search recommendation advertising will be fully cloud

Nowadays, Alibaba Group’s business, especially the comprehensive cloud of search recommendation advertising (search promotion) business, is of great significance to the whole Ali Cloud, especially the storage team.

At the beginning when doing the ali cloud, jian wang academicians have a goal, the dream is ali cloud can support multiple search engines, including not only its search business, ali also hopes to have the external search company architecture to ali cloud above, this means that the cloud platform needs to have the corresponding ability to support search this technically complex heavy business. “In fact, the process of search marketing cloud a little twists and turns, we from cloud to cloud to cloud, after 12 years of development to come this far, but in the end, we prove that this thing is set up in 20201, and we have all on the cloud, so it is in the whole development of storage technology a very gratifying achievement.”

Current storage challenges and interesting findings

Current storage technologies have made great progress, but some technology trends bring new challenges that need to be addressed through full-link design and architecture reconfiguration.

The first is stability, a perennial challenge. In the past, the usability of the system might have been three nines, but now it’s four nines, five nines, and we’re not satisfied with that, we’re looking for better stability metrics. Due to the introduction of new equipment, new business, new load, new technology, the architecture of the whole system is changing, there are higher requirements for the bottom. For example, cloud native brings fast and elastic pull, and without local storage memory acceleration, the bottom layer will take a huge hit.

In other words, is the physical capacity of the disk perfect for storage virtualization?

“Even though disk capacity has become very large today, if you divide performance by capacity, it’s interesting that disk performance per capacity is actually declining.”

Jason gave us an example: “For example, you might have had a 4T disk, and a few years ago it went to 8T, and now it might be 20T. Other dimensions of HDDS are constant. The IOPS ranges from 120 to 150 and the bandwidth ranges from 150 to 200 mbit/s. But disk capacity has increased fourfold from 4TB to 20TB, but neither of the other two dimensions has changed, so if you divide those performance measures by capacity, you see that the whole curve goes down.”

What happens if a 20TB disk is shared with more users in a multi-tenant way through resource reuse and cloud technology?

“I give each client 4 TB for a 20 TB disk, which is equivalent to five 4 TB disks, but each of my clients expects 120-150 IOPS. But the physical capacity of the disk is not enough to support the 1-to-5 capability of storage virtualization, so today it is a big technical challenge for us to solve.”

In the case of multi-tenant, there is also the need for resource performance isolation, how to do a good job of non-interference, how to use the full power of the hardware, how to reduce the cost. These are classic problems in the storage field, but in today’s large-scale cloud environment, all the problems are magnified.

Ali cloud storage evolution direction

In the face of the above challenges, what is the next direction of Ali Cloud storage team? In addition to further improving the usability of the system, Jason revealed that they will face new load evolution such as data lake, machine learning, AI, autonomous driving, search promotion and so on. First, they will release a faster distributed file system based on Pangu, which currently serves alibaba group’s internal search promotion business. The second is to continue to do a good job in parallel file system CPFS, which has been applied in AI computing and automatic driving scenarios. It is worth mentioning that “Pangu”, a distributed storage system developed by Ali Cloud, was selected as the world’s Leading Internet scientific and technological achievements last year. In addition, they will also cooperate with other aliyun products to provide customers with high-quality data management and data governance capabilities. It also focuses on the continuous evolution of high-performance networks, self-developed network cards and disks, and the integrated design of hardware and software between networks, disks and servers.

“In fact, we have two main battlefields in the future, one is to deepen the foundation, continue to innovate in chip disk, network and server design, looking for the opportunity of full stack multi-point technology integration, the other is to do product design for users. For example, when Alibaba Cloud and AWS designed EBS products in the past, each disk was given a fixed performance index. If users wanted to change to another performance index, they had to perform the change operation, which took several minutes or even more than ten minutes, greatly affecting user experience. Therefore, Ali Cloud launched ESSD Auto PL after redesign, that is, on the set bottom line, can be infinitely variable speed; In other words, users can rise and fall freely based on their own service requirements without too much consideration of capacity resource allocation. “This design is a very challenging thing for the product and research and development, but it also confirms Aliyun’s own culture of keeping the complexity for ourselves and leaving the simplicity to others.”

Personal growth ramblings

Beyond the technology, Jason shares a few of his own stories.

Talk about time management

With teams in Seattle, Silicon Valley, and China, Jason had to travel back and forth between China and Seattle frequently. Before the epidemic, Jason would come back four or five times a year for one month each time, but after the epidemic, he would come back once a year for three or four months.

We are curious about how Jason manages his time and works efficiently under such circumstances. Jason says he started out as a programmer and then became a manager. When I was a programmer, my time was relatively continuous and concentrated. In addition to writing code and debugging code, I had to take part in some design tests at most. However, after I did management, one of the most obvious changes was the increase of meetings, and some important meetings had to be attended. In addition, there are some time costs that don’t show up on the calendar, such as spending more time with other students on the team and communicating across teams. “I now have a small goal of communicating with at least five students a month who don’t report directly to me, so you can get to know some of their ideas and get to know the team.”

“Part of it is that it takes a lot of effort to find like-minded people for the team or for the field, take the time to talk to them, share your ideas, and wait for the right opportunity to bring them into your team.”

2. Talk about pressure

In his busy work, Jason is facing a lot of pressure, the most important pressure is how to ensure the stability of the system and provide customers with a stable and reliable operation state. This may seem simple, but it can be very difficult to implement, especially in very large distributed systems. “When there is a jitter or glitch in the system, all kinds of unpredictable things happen and the customer’s business can be damaged, how can we quickly recover the system and minimize the impact on the customer? A lot of times it’s a race against the clock, and that’s my biggest pressure right now.”

When it comes to whether there are KPIs, Jason said his KPIs are very simple — the system has no P1 failure, is always online, and the customer’s data is not lost on Ali Cloud.

“The foundation is not strong, the earth shakes”, Jason mentioned this sentence many times in the conversation.

3. Talk about decompression

Jason will relieve his pressure from both physical and mental aspects. Usually, he likes some sports, such as football. In Seattle, he plays twice a week, usually at night, and Friday night falls in the early morning hours of a Saturday in the country, which can be staggered. As he got older, he moved down the ranks, playing from striker to centre-back in addition to goalkeeper. “It helps your body to be in better shape and cope with more stress, so at least you need to be in good shape to stay up late and look for problems.”

In terms of mentality, Jason suggests looking at the problem from the perspective of development. When a fault occurs, instead of blaming people in the first place, we should optimize the system, process, team cooperation and so on to avoid the occurrence of the next fault. , and then check whether there are similar vulnerabilities in other modules and systems for improvement. Jason told his team, “A strong product, or a very strong system, a lot of times it grows with failure.”

Systems grow by making mistakes, and so do people.

“But every fault the impact to the customer, we are also very pain, we think these things I can do better, so in such a process of constant relative spiral, we make the whole system can achieve what we want that state is never stop, I feel this mentality still is to relieve stress is helpful to us.”

4. Talking about their study

In his spare time, Jason also arranges some study time for himself. He pays attention to what papers are published at USENIX, OSDPI, SOSP and other top conferences, and what engineering blogs are posted by Companies like Facebook (now called Meta), Twitter, Amazon and Google. “I also read a lot of InfoQ articles, geek hour, and I think InfoQ is a great community, IT hub. Sometimes, I also read some q&A on Zhihu, which I think is very good.”

Some cutting-edge technology trends and even the focus of national innovation can be found in the top international conferences. We also see a positive phenomenon, at present, there are many domestic teachers and graduate students, including Ali and other enterprises, published more and more high-level articles in these top conferences, which also reflects the rising level of research in our country.

It is understood that Ali has funded several lectures of excellent papers in the field of storage. Jason told us that although the effect is slow, academic research is definitely worth long-term investment, because in many cases, a theoretical breakthrough will lead to greater development of the entire industry. Dharma hospital construction is invested in the medium and long-term technology research on ali, ali and cloud storage team itself also doing some technical preparatory work, cooperation with many colleges and universities every year, recruit interns, on the one hand, hope to give college students a real working environment, the industry at the same time also hope that through some of the joint research project, To give teachers and students a deeper understanding of industry and measure solutions to problems. The more institutions or organizations like Dharma Academy, the better for industrial breakthroughs.

5. Talk about your sense of accomplishment

Jason: I think engineering and management might be similar in terms of the sense of accomplishment, but in different ways. The sense of achievement of being an engineer comes from the system being online and running stably and serving many customers. As a manager, it comes more from team development, you see the students on the team grow up. Because he is in charge of the storage business line, he also has a sense of accomplishment from customer feedback. “The customer thinks this product is good, and I want to recommend it to others, so I get a greater sense of accomplishment.”

6. Talk about choice

When talking about big decisions in the past, Jason told us that he had some confused moments, but not many. “In fact, most of the time you are like a scale, as long as you add a straw on either side, you may break your confusion.”

Jason: Well, I’ve changed my job. His first post-phd job in ASK, Base was in sunny Silicon Valley, but moving to Microsoft would have required a move to Seattle, with its rain and overcast days, so he was hesitant. But the thought that housing prices in Seattle were at least half as low as those in Silicon Valley, “I found some psychological balance.”

7. Talk about goals

In terms of work goals, Jason has made OKR for himself, among which an O is P1 failure free, and some very ambitious goals. “Some of our goals are real goals, and some of our goals are traction.”

Jason admits that some of his personal goals haven’t been met. In 2021, he set himself a goal of running 365 miles, but he only managed about 100.

“I said that I would communicate with at least five students who did not report to me every month, but I think I have not done so well, about 60 or 70%, which means that there are only 3 or 4 students but no more than 5 students every month.”

Jason believes that the goal of team development, especially overseas team building, has not been particularly met. On the one hand, due to the impact of the epidemic, on the other hand, due to the current technological trend, the competition in the whole overseas talent market is very fierce.

In 2022, Jason hopes that the epidemic will gradually end and he can spend more time back home. In addition, he also hopes to spend more time on vacation with his family.

8. Talk about growth

In 2022, the theme of Geek-Bang is “infinite growth”. Jason interprets this word from three dimensions of personal growth, technical growth and non-technical growth.

In his opinion, no matter in the encounter of not good at things, can take the initiative; “Or to take the useful things from all kinds of feedback to make yourself better; , or not limited to the existing ideas, do product innovation, to bring more value to customers, this is infinite growth. This is similar to the idea of “breaking boundaries” advocated by Alibaba. “From a collaboration perspective, if you build something, you can not only use it yourself, but you can promote it to other people, to other products, to other teams, to make your module or code more valuable.”

“In fact, if you look at the development of mature technologies, the entire team is actually on a different scale. For example, with ESSDC, we were the first block storage team in the industry to come up with a 100 μ s(Microsecond) delay and set a goal of 1 million IOPS. In 2018, we launched ESSDC, and we achieved that goal, and we’ve been leading the way in performance so far. This year we set ourselves a target of 80 delays, and we have a new product that can do 30.”

This pursuit of technology also encourages developers to break the status quo and make innovations, so that they can grow with the technology trend in the industry and turn their technical goals into products, so that products can be more competitive and serve more customers. Infinite growth is also reflected here.

Non-technical aspects of growth, including how to do project management, communication with the team, improve the team cooperation ability, to become the driving force of team effectiveness. “We often say in Aliyun that we should fight together and win together. I think our culture, whether it’s my team or whether it’s ali Cloud’s culture, technical culture or organizational culture, actually fits in well with infinite Growth.”

conclusion From the conversation, we found that Jason is a very simple engineer, as thousands of lives in the field of basic software practitioners, their efforts perfusion from generation to generation solid underlying infrastructure, become the strong support of the top business, make enterprise in the wave of the digital transformation to meet the challenge of the unknown.

His growth experience also confirmed our point of view — the bottom is not necessarily the top, domestic basic software field is promising. Basic software has long been an area of focus for InfoQ, and the deeper you go, the more you see the value behind home-grown basic software, which is why the DIVE Global Basic Software Innovation Conference was born.

The original link

This article is the original content of Aliyun and shall not be reproduced without permission.