On June 5, 2021, the 2021 China Developer Ecosystem Summit hosted by SegmentFault came to a successful conclusion. At the meeting, Yubo Wang, chief evangelist of Amazon Web Services, delivered a keynote speech on the integration of cloud, open source and community from the practice of Amazon.
Guest: Yubo Wang, Chief Evangelist of Amazon Web Services.
Sort and publish shorthand: SegmentFault Editorial Department
Hello everyone, today my share is about some of Amazon’s contributions and practices in the open source field.
I believe you will have some understanding and consideration of open source. So today I want to talk to you from the perspective of cloud computing vendors, how we are enabling more developers to use open source software in the cloud faster, easier, more secure, and to help them with some applications and deployments.
In fact, whether for our customers, or partners, or developers, a lot of people are concerned about how we can use open source software faster and better. First of all, only by applying our open source tools well can we consider how to make further contributions to open source and how to make some contributions to the community in the subsequent practice.
Who cares about open source?
As a cloud computing manufacturer like Amazon Cloud Technology, it has actually done a lot of work behind the scenes. For developers, there are many reasons to be interested in open source, including personal interests, personal brand promotion, and career development. In fact, open source has a lot of appeal for different developers, and many developers are motivated to make a lot of contributions around open source and the community. We have seen many developers gain influence through open source in their careers, which has enabled them, their companies and their communities to continue to grow and thrive.
For enterprises, there are also many enterprises are doing further open source. I have here a set of data that Black Duck did a few years ago. Currently, 96% of the software used in enterprise environments contains open source code or projects. Open source has become an increasingly important part of the core software architecture of the enterprise. And the use of open source can also enable our enterprise to carry out rapid development, rapid practice, rapid iteration, can quickly make our products to the market for our customers to make better use of and better help, so that our enterprise innovation can be further improved.
From the perspective of developers and enterprises, we can see that both enterprises and developers are constantly using and embracing open source. For cloud computing manufacturers, we can see that many cloud manufacturers, including Amazon Cloud Technology, are actively embracing open source. In fact, the convergence of cloud and open source, cloud and community is also a very big trend. Here I would like to quote a group of data, currently in the world’s top five open source vendors, four of them are cloud computing vendors. Seven out of the top ten. It can be seen that with the continuous development of cloud computing and open source, more and more cloud computing manufacturers are also embracing open source, so that we can enable our developers to use services more and more quickly, conveniently and safely on the cloud.
For our customers and our developers, what do they think of the open source leadership of cloud vendors? Here is a set of surveys we did:
In fact, this group of survey is not only for Amazon Cloud customers, but also for many customers who do not use Amazon Cloud technology services. From our survey results, we can see that more customers also hope to deploy open source software quickly and conveniently in the cloud computing environment, just like what we talked about before, so that open source software can quickly bring practical results for enterprises and individuals to improve production efficiency. Of course, many of our developers are looking to cloud vendors to provide them with a range of better ways to use open source, and to contribute more to open source through cloud computing.
In fact, from the perspective of developers and cloud vendors, their goals are basically the same, both hope to make our open source software fast and convenient to use in the cloud environment.
Amazon Cloud Technology, the world’s leading cloud computing company, has been focusing on the open source field. Of course our voice in the open source space, compared to some of the other vendors, doesn’t seem to be a very strong voice. But we do a lot of work behind the scenes to help our customers, to help developers use and deploy open source software in the cloud.
Amazon has a very important leadership principle called customer first, and we do a lot of innovation on behalf of our customers. We’ve made it easy to deploy open source software in the cloud that’s very difficult to scale in production. This enables our developers to quickly and easily build an open source environment that can be expanded rapidly through simple cloud services in the cloud. At the same time, we at Amazon Cloud Technology have also initiated a series of open source projects to help developers solve their practical problems in the production process.
We’ve also built a whole community system through us, community evangelists, and friends we call ‘heroes’ to push the convergence of cloud and open source as well as the expansion of the community. We also have a lot of employees involved in open source within the company, and we encourage them to contribute and participate in open source software. Of course, some are project-related, and some are personal interests of the employee. At Amazon, the number of engineers contributing to open source is increasing year by year. The number of open source repositories we contribute to at GitHub is now over 2,500, covering areas ranging from basic computing, containers, serverless to data, artificial intelligence, machine learning, robotics, and more.
We continue to contribute to the open source community in many areas. So here are some examples of open source projects that Amazon has been involved in, just a few of the projects that we’ve been involved in.
We have contributed to and participated in many open source projects, and we support a range of open source technology projects in different ways. For example, through a series of financial support and the use of cloud services, help the community support to help our developers, rapid application in building up in the clouds, at the same time we also through a series of recruitment, development projects to increase our for developers, support for the builders, makes the whole amazon cloud technology to build a system of open source developers.
Amazon cloud technology’s contribution to open source is mainly concentrated in two aspects. On the first hand, we hope to provide a series of cloud services through Amazon Cloud Technology to help developers quickly, safely and conveniently use open source software on the cloud. Second, we have launched a number of new open source projects to solve the real problems of our customers. This chart illustrates a series of efforts we’ve made towards the first area, to show you how to quickly implement open source builds on the cloud.
It is known that Amazon Cloud currently offers 200 full-featured services, many of which are open source related. We help open source software to be installed, used and deployed on the cloud, which makes it easy for our developers to develop and practice in the cloud. For example, in the cloud, we have easy access to software like Cassandra, FreeRtos, Kubernetes, etc., and we have added a lot of observability, manageability, and security monitoring to it, which allows us to use open source for easy production and deployment in a production-oriented environment. In addition, we have also packaged and released many new open source projects in the form of cloud services, so that more customers can quickly experience the advantages of open source in the cloud. For example, Lambda, which is based on Firecraker, is our leading server-less computing capability, and it enables the latest open source capabilities to be better reflected in the cloud service.
What open source projects do we participate in and contribute to?
The first, of course, is Rust, a very popular programming language right now. We have been sponsoring the Rust Open Source Project since 2019, and many of the major contributors to Rust currently work at Amazon Cloud Technology, including Felix Klock, one of the lead compilers we recruited to the Rust team last year. He also joined Amazon Cloud. Many of our cloud services, including Amazon S3, Amazon EC2, Amazon Route53, are starting to build more with Rust, and we’re building a lot of new services with Rust. Through a series of staffing and product enhancements, Rust products will be used in the cloud while also providing additional assistance to enrich the community. For developers, we launched the Amazon Cloud SDK for Rust this year to make it easier for more Rust developers to use the native Rust SDK for cloud native application development and practice. This is a series of contributions and uses that we have made in the Rust field.
The second area is graph computing. So graph computing in the open source world has Apache TinkerPop, which is an open source graph computing framework. Currently, our engineers at Amazon Cloud Technology are PMC Chair at TinkerPop, which provides a lot of energy for the development of the community, and we continue to interact with the developer community. We have engineers who have been consistently evangelizing for several years on Graph Computing in the community, including Practical Gremlin online resources to help more developers use Graph Computing framework TinkerPop. At the same time, our Amazon Cloud Technology also launched a cloud fully hosted service, called Neptune. It is compatible with open source TinkerPop and provides a very easy way to build graphics applications in the cloud with the help of cloud services. We also collected many developers and domain of product feedback, we are also in our cloud services through constant feedback to our open source community, errors related to repair, strengthen related functions at the same time, further to the open source community through the cloud feedback related products, which makes the interaction between the cloud and open source community more and more, better and better.
The third area is search. Over the years, we’ve used a lot of open source technology, whether it’s Amazon e-commerce, whether it’s search, Apache Lucene is one of the many search technologies that we use, and now on Amazon e-commerce, or in Alexa voice, Many search technologies are built using open source Lucene. Of course, we’ve added a lot of new features to the Lucene build, including how to make searching more efficient in the complex case of large-scale concurrency. We also continue to update the code and make low-level changes to Lucene at this level, making Lucene an ideal choice for heavy work. In addition, we have added a number of new features to the Lucene open source platform, such as the ability to rank behavior signals, which is how to proceed once a product is found, which has been a focus of the Lucene community for many years. Therefore, our ability to provide the Lucene community with something called indexing the frequency of custom terms allows us to further enhance our contribution to open source and make it more accessible to different developers.
The fourth area is Redis. We’ve also contributed to Redis for many years, and currently one of the five Redis maintainers is working at Amazon. In Redis, we contribute a lot of encryption-related features. Many years ago, Redis itself did not support encryption, when Amazon engineers came up with PR, hoping to add a series of TLS, including SSL encryption, to Redis. We added these features and got them approved, making Redis even more secure. Through our combination of open source and cloud services, more and more companies are able to further integrate open source and cloud services to help them deploy Redis solutions on a large scale, such as our typical customer Airbnb, which has a very large Redis cluster itself. But moving from a homegrown data center to a cloud service actually has several different processes. First of all, he migrated from the local data center to the EC2 platform, hoping to make use of the management experience of local engineers to migrate to the cloud and take advantage of the elastic resource expansion advantage of cloud services to expand the cache architecture. But as the cache cluster expanded, it found that the operation of managing the configuration of the cluster was a time-consuming and labor-intensive process. As a result, he eventually migrated all of his caching applications to a managed service called Elasticache. By hosting services, you can not only use the advantages of many open source projects, fully compatible with open source, but also greatly liberate productivity. The management and maintenance engineers no longer need to install, maintain and manage the underlying architecture, which can free up energy to do better in the upper level Redis applications. It is also an important demonstration of how the cloud can help customers further advance in the open source field.
Of course, that includes Amazon Hudi, which is itself a feature of Amazon EMR in our hosted product. Currently our engineers are also members of its PMC. We also contribute a lot to Hudi, including Hudi boot support, Hudi file listings, query planning, and more. With Hudi, the ability to read S3 file lists is greatly improved.
For the robot industry, ROS is a very important robot software. Amazon Robomaker, the development framework, will help many developers to expand ROS in the cloud and facilitate them to build better and faster robot applications. Amazon is currently a member of the ROS Technical Steering Committee and the relevant working group on ROS 2. In addition, ROS 2, the latest version of ROS, has contributed many new features, bug fixes and usability improvements, as well as more security features, so that developers can develop robot-related applications quickly, easily and safely.
In the field of Kubernetes, we ourselves are the platinum members of CNCF and have made contributions to many projects of Kubernetes, such as the container network interface CNI in Kubernetes. In SIG Security, we are the second largest contributor. In addition, we have contributed a lot to security, observability, and manageability. Kubernetes has been deployed in many production environments. However, there are many things to consider for the deployment and construction of Kubernetes environment in the production environment, not only some functional features and improvements that are usually paid more attention to, but also security and management, which are very important topics.
As a result, we have greatly enhanced these security management features in the hosting service Amazon Eks, as well as feeding back a number of security features into the community, which allows us to develop code and features that can be used by many Kubernetes customers. This series of enhanced security and management-related updates is available to Amazon Cloud hosting services, other cloud vendors, and local environments, helping customers better deploy their development in production environments.
We have a lot of contributions in other areas, including the network area, the automated distribution collection area, the measurement area, we have a lot of contributions as well. We also have a lot of innovation in Kubernetes. For example,
- How to run a Kubernetes cluster on an ARM-based architecture and optimize cost
- How to deploy and invoke a fast machine learning model in a Kubernetes cluster
- How to interoperate with a range of VMs, GPUs and Kubernetes cluster compatibility plug-ins
We’ve done a lot of coding in these areas as well. Through a series of continuous updates to Kubernetes, the ease of use and security in Kubernetes have been further improved, and Amazon cloud technology has gained the core leadership in the field of Kubernetes.
We just talked about the first part, how we can use cloud services to accelerate the use, installation, deployment, and management of open source software. Below, let’s take a look at some of the open source projects that Amazon has started.
Many of our open source projects are not imagined or invented, they are based on customer needs. But sometimes customers only know the goals of the vision and do not know exactly what the relevant open source products are. Therefore, Amazon will innovate on behalf of customers and launch a series of new open source projects to meet customers’ growing demand for open source products. We have initiated open source projects in a wide range of areas, from databases and data analytics to machine learning, from computing to front-end development, from serverless to development tools to security and more. We have launched a number of open source projects in a variety of areas to help our developers solve more practical problems. It also invites developers to engage in further interaction to drive the convergence of cloud computing, open source and community.
One of the newest projects is called Babelfish for Aurora PostgreSQL. This is a new open source project released by Amazon Re :Invent 2020 at the end of last year. It actually solves the problem that many customers migrate from the traditional commercial database to the open source database in the cloud. As you know, migration is a very complex process. In addition to migrating the underlying database, it is also important to consider that the upper-level application can be migrated once and for all with little or no code change. Babelfish is such an open source product. Babelfish provides a transparent transformation layer to facilitate application migration to PostgreSQL, enabling customers to use open source databases faster and in the cloud, reducing costs, increasing ease of use and making new contributions to the open source community. The open source project will be officially available on GitHub later this year, among other projects.
The first is Amazon Corretto, the JDK. It is well known that Java currently has a number of issues in the licensing space. So we launched the Amazon Corretto product to participate in the OpenJDK community. We are also currently a leader and partner in the OpenJDK community. We also have many new capabilities on top of Amazon Corretto, including increasing the performance of encrypted things on Java, optimizing the ARM architecture to help developers build better commercial applications using the open source JDK, or open source applications, etc. We also continue to provide an unlimited number of free security patches for Amazon Corretto. We also have a lot of investment in Java. James Gosling, the father of Java, is also currently a Distinguished Engineer at Amazon Cloud Technology, responsible for many product-related developments. A range of open source software, including Corretto and even Greengrass 2.0, also released late last year, have provided invaluable guidance for a range of open source development.
In fact, for open source projects, personnel is always a very important factor. Amazon Cloud Technology also attaches great importance to the recruitment of talents in the open source field. Many of our engineers and partners are very, very excellent open source experts and talents.
Another open source project is Firecracker. It is an open source virtualization technology under the Apache2.0 license and is compatible with a variety of capabilities. One is containerized security management capability, and the other is the speed and performance of the container. Before Firecracker, in the field of container, it is very difficult to have both sides of Firecracker. However, with Firecracker, it can give consideration to two aspects, which provide very good help for container virtualization, including the application without server. Just now, as we mentioned our cloud services, we also use many kinds of Firecracker related technologies, so that people can directly use our open source products to build applications for Firecracker.
For Firecracker, we do not just regard it as an open source Firecracker, but hope to build a complete ecosystem including developers and partners, so as to facilitate the community and developers to use it quickly and conveniently. For example, in North America we have a partner called Weaveworks. It’s also a very well known startup in the container, serverless space. They have done a lot of development based on Amazon’s open source products. For example, he used the eksctl tool he developed, which is now our official interface to Amazon Eks. The Ignite product based on Firecracker is built to further carry out the development and management of server-less and container-oriented Firecracker. At the same time, through ecological cooperation, the scope of use of developers has been expanded, so that our open source products have better iteration, and at the same time, our circle of friends has been constantly expanded.
Then there’s our Bottlerocket in the Linux operating system space. BottleRocket, an open source Linux operating system built specifically to further enhance the system’s security and flexibility, helps developers handle different transactions. In the Linux field, we have contributed a lot of open source products, including BottleRocket and Nitro Enclaves, to achieve more secure virtual isolation, and to provide better security experience for various industries, such as the financial industry and some typical data sensitive areas. It enables hypervisor management of virtual machines across multiple operating system and processor architectures, and it builds on open standards and interfaces that can be built across a range of environments for environment reconstruction and deployment.
We also want to provide more tools for developers. On top of CloudFormation, a new tool called CDK, the Cloud Development Kit, is 100 percent open source and can help developers leverage infrastructure-as-code for dynamic deployment and resource reuse. A number of tools are available to help developers view and use distributed code from source. At the same time, the project was actually contributed by a large number of engineers, especially engineers in the community. Forty-eight percent of the project’s contribution was community-driven, enabling the product to be recognized by many community partners from the beginning of its design. Also makes the product in many developers have been very good recognition.
And then there’s OpenSearch, which is an open source product that actually just launched last month, based on Apache2.0, an open source search and analytics suite. Elasticsearch changed its license from Apache2.0 to SSPL earlier this year. This is a lot of developers using ElasticSearch in the cloud. It actually provides a very unfriendly protocol experience. So we decided to re-use Apache2.0 to build a suite of open source software. So we forked Elasticsearch and continued to use Apache2.0 for related open source distribution and deployment. At the same time, we’ve added a lot of new features and core capabilities. OpenSearch is now open source, and we hope that our community and developers will join us to provide a better environment for the growth of a more OpenSearch platform.
For a range of observables, we’ve also launched our own distribution, Open Distro for OpenTelemtry. To further add a series of related functions to the upstream observability, provide a series of API and SDK specifications. At the same time, it provides many contributions to the upstream.
Amazon Cloud Technology is CNCF’s main partner. It has collaborations on many levels, including Kubernetes, Prometheus and Envoy, to name a few. At the same time, we also hope to bring more CNCF open source projects to the cloud environment to provide more possibilities and convenience for developers to use. At the same time, we have also made a lot of contributions to CNCF projects, including ETCD Nanny, CDK8S and so on. At the same time, we are also continuing to contribute to some CNCF upstream projects. I just talked about, including a lot of contributions to Kubernetes.
Amazon Cloud has made a lot of contributions in the open source field, and our driving force is actually from the end customer. We adhere to the concept of customer first, customers want to use open source software quickly and conveniently in the cloud environment, customers want to have product innovation in the open source field. As a result, we are innovating on behalf of our customers to launch more open source projects, and more open source cloud services. Being customer-oriented allows us to get closer to the market, listen to developers, and create better cloud products and open source projects to give back to everyone.
We have a lot of open source contributors internally, and I’ll just give you a couple of examples, and you can see that a lot of our open source contributors are also women. Through the efforts of a series of developers, Amazon’s open source contributions have grown larger and larger, attracting more developers to participate and contribute to the community.
In addition, we continue to help the open source community grow. Drupal is a very popular community, currently supporting over 550,000 websites, over 1.3 million communities, and many developers. It’s currently built on top of Amazon’s cloud technology platform, and we’ve contributed a lot to the thriving Drupal community. Back to more than 20 years ago, when he started his business, it was built with open source MySQL, but with the increasing number of users, he found it very difficult to expand MySQL. There are a lot of options to adjust, a lot of communication between nodes to take into account, and a lot of extension details. So in the process of development in practice, they began to use amazon cloud the service of science and technology, through the cloud service fast elastic scaling characteristics, can greatly save the cost, at the same time for elastic capacity improvement, makes the combination of cloud services and open source software to support up to millions of users at present, the community and participants in the tens of thousands of users huge ecological platform developers.
We also continue to help open source companies grow. You know, Amazon has a third party Marketplace called Marketplace. Many third party open source companies can put their open source products on our Marketplace. We can work closely with open source companies to help them get further into the market. Because whether it is commercial software companies or open source software companies, especially the operating companies behind open source software, certainly need to have a way to make money. Therefore, we hope to make the open source market bigger and stronger, and to share the fruits of this with many partners. The Marketplace is a great place to collaborate, making it possible for cloud and open source to join forces.
Of course, we also help many customers build a range of best practices for open source projects. Lyft, for example, is also known to be doing a lot of work on autonomous driving. It’s called Envoy, which is built on top of Amazon’s cloud technology. Amazon engineers have been working with them a lot to help them quickly extend Envoy on EC2, including Kubernetes. We’ve done a lot of contributions with the engineers at Lyft, including Netflix, which will have its own open-source Spinnaker product. During the collaboration with Netflix, our engineers constantly found errors in Spinnaker, so our engineers also participated in the open source project and made a lot of code contributions and performance improvements.
We also have a lot of funding to help a variety of foundations support open source and community growth. We also hope to have more products to help customers use open source. Migration from traditional commercial licensed software to open source licensed software. For example, the Database Migration Service Service helps with Migration to open source databases, and so on. Whether you’re using open source cloud services or building open source on your native infrastructure, we provide tools to help you move quickly to open source.
We also have a series of stages for developers and the community. Amazon cloud technology also has expert projects for developers, which have two levels. The first is Hero, which is relatively few in China at present. We also hope to work with many developers to quickly build an ecosystem in China. If you are good at technology and a certain field and want to speak publicly on more occasions about how to use cloud service products, build open source or use open source to implement solutions, you can sign up for our project and constantly improve your influence in our activities. If you think Hero is still a bit difficult, we have a second project called Community Builder. Builders in the community and everyone to carry out a wide range of communication and interaction, for everyone to share knowledge, provide the guidance of relevant technical resources, drive the vigorous development of the community.
For cloud services and open source, in fact, everyone can see the current trend of convergence and development. Cloud computing is further driving open source, and open source is allowing more and more developers to use the cloud to build applications. Cloud and open source, in fact, is a non-zero-sum game. Our common hope is to make the market bigger and stronger, so that more open source projects, open source manufacturers, and more cloud service providers can find a suitable position in the growing market, and provide more choices for developers. Therefore, we also hope to work with more Chinese developers to make open source better and stronger.
Finally, I want to say that open source is not only Chinese, it is global. It is hoped that developers can stand in a global perspective, to look at open source and contribute to open source. We hope that through our technological influence, developers all over the world can benefit from the contribution of Chinese developers, and we also hope that more and more Chinese developers can expand their influence to the world.
Thank you very much.