Recommendation of the article: “In the just ended year of 2020, I had the honor to participate in a community activity of AWS. In this activity, I heard Cloud Computing, Open Source and Community shared by Yu Bo for the first time, which brought me a lot of thinking and inspiration. After the event, I invited Yubo to write an article to share some content related to AWS and open source with developers in the community. Yubo took time out of his busy busy to write the following article, which will be of great help for developers to understand AWS’s participation in open source and which open source technologies are used in the latest products. We also welcome developers in the community to participate in the community activities of AWS and Synoptec to share cloud computing and open source technology.” — Gao Yang, CEO of SegmentFault.
Photo taken at AWS Community Day Shanghai 2020
Wang Yubo
About the author: Yubo Wang is a senior developer evangelist at AWS. He is responsible for evangelizing and promoting AWS technology and products to developers. Prior to that, he was a Senior Product Manager at AWS for many years, where he was responsible for business development of AWS computing, data, and artificial intelligence-related products and solutions. Prior to joining AWS, he held product, technology and management positions in several multinational companies and has nearly 15 years of experience and practice in the IT industry. He also has many years of research experience in the fields of computer vision, pattern recognition and machine learning.
Viewing AWS’s new contributions to open source and the community from Re :Invent
Since offering its first cloud service in 2006, AWS has been a major player and supporter of the open source community, contributing to code, operations, infrastructure and more. In addition to the more than 2,500 GitHub repositories we’ve built, we’ve been involved in thousands of third-party open source projects and nurtured an open source community of hundreds of thousands.
Figure 1: AWS contribution to open source is increasing year over year
As a cloud computing vendor, AWS has taken a more balanced and pragmatic approach to open source, attracting many employees, customers and partners to participate in and contribute to open source projects. Our driving force in the open source space is the end user, focusing on providing better tools for developers. When we release cloud services based on open source projects, we make a long-term commitment to support our customers, providing open source projects with bug fixes, security, scalability, performance and feature enhancements, and an “upstream first” strategy that enables us to work on Linux, Xen, Kubernetes, Redis, Rust, Redis, GraphQl, Ros and other communities have won leadership positions. At the same time, we are also engaged in innovation on behalf of our clients and have launched a number of open source projects such as Apache MXnet, Firecracker, Bottlerocket, Partiql, Amazon Corretto, Open Distro for Elasticsearch, AWS Distro for Open Telemetry, etc., make it easier for developers to transfer from proprietary to Open source and contribute. In addition, we are actively involved in customer-initiated open source projects, such as Netflix’s Spinnaker. We also continue to work with some of the leading open source ISVs, including MongoDB, Databricks, and Redis Labs, to help them build and optimize cloud services.
Figure 2: Some of the open source projects that AWS is involved in
Figure 3: Some of the open source projects initiated by AWS
In the just-concluded AWS RE :Invent 2020, there are a number of new contributions to open source and the community, among many new releases. Here we focus on DevOps, data and analytics, and machine learning and edge computing, which developers are more interested in.
First let’s take a look at DevOps.
When it comes to DevOps, containers and Kubernetes are the next generation of infrastructure for the DevOps tool. Amazon Elastic Kubernetes Service (EKS) provides a secure, reliable, hosted Kubernetes cloud Service for many developers, but there are also many developers who have to run Kubernetes locally for a variety of reasons. To help those same developers avoid the heavy lifting of managing the Kubernetes infrastructure, we’ve released the Amazon Eks Distro, an open source database. It is the Kubernetes distribution used by Amazon Eks and helps you manually run a Kubernetes cluster anywhere, including on Amazon EC2, in other clouds, and on local hardware. Eks Distro includes upstream open source Kubernetes components and third-party tools, including configuration databases, networking and storage components needed to create clusters. There are Kubernetes control plane components (Kube-Controler-Manager, ETCD, and Coredns) and Kubernetes work node components (Kubelet, CNI plug-ins, CSI Sidecar images, Metrics) Server and AWS-IAM- Authenticator).
At the same time, we also offer Amazon Eks Anywhere a brand new Kubernetes deployment option. Amazon Eks Anywhere builds clusters based on Amazon Eks Distro, making it easy to create and operate Kubernetes clusters locally, including on your own virtual machines and bare-metal servers. Eks Anywhere offers installable software packages to create and operate Kubernetes clusters locally, as well as automated tools for cluster lifecycle support.
With these two new releases, it can be said that in the Kubernetes space, AWS offers developers the same usage and operating experience across both local and cloud.
Figure 4: The same operating experience for Kubernetes across the local and cloud
Continuous observability is a key component of the DevOps pipeline. The three pillars of observability — tracking, logging, and metrics — need to act as a unified, integrated set of tools that allow every developer to view all observations in one place. In Re :Invent 2020, AWS has introduced three new open source solutions to integrate better observability in the cloud. First up is AWS Distro for Openelemetry, a secure, production-ready, AWS supported Openelemetry project distribution for applications running on EC2, Lambda and AWS container services, as well as native applications, Send metrics and tracking to multiple cloud native and open source monitoring components. The second is Amazon Managed Service for Prometheus (AMP). Prometheus is a popular open source monitoring and alerting solution optimized for container environments. While it is easy to deploy a single Prometheus server on AWS, it can take weeks of manual work to scale across multiple servers and configure the environment for high availability. Customers who need more capacity must deploy other open source projects, such as Thanos and Cortex, and put in more engineering work to optimize resources such as memory and storage to control costs and optimize query re-response times. With AMP managed services, you can use the same data model and query language as the open source Prometheus to monitor the performance of containerized workloads and gain improved scalability, availability, and security without having to manage the underlying infrastructure. Amazon Managed Service for Grafana (AMG). AMG, based on the open source Grafana project, is a fully hosted service with rich interactive data visualization that helps customers analyze, monitor and alert on metrics, logs, and tracking across multiple data sources. You can create interactive dashboards and use auto-scaling, highly available, enterprise-grade security services. At this point, AWS has provided a complete cloud native and open source continuous observation solution in its cloud services, giving developers more options to build DevOps.
Figure 5: Cloud native and open source continuous observation solutions
Next, let’s look at the data and analysis.
In the database space, PostgreSQL has always been a popular choice for developers. One of the reasons many developers like PostgreSQL is that it is entirely community-driven. But migrating to PostgreSQL is not easy, because the biggest barrier to database migration is inertia. Whatever the reason for migrating to PostgreSQL, developers want an easier way to do it. With this in mind, AWS launched the open source Babelfish for PostgreSQL in Re :Invent 2020. BabelFish is PostgreSQL’s Microsoft SQL Server compatible endpoint to enable PostgreSQL to become proficient in understanding database requests for SQL Server applications, including understanding T-SQL and TDS communication protocols. With Babelfish, there are few code changes and no changes to the database driver to help more developers migrate to PostgreSQL easily. We will officially open source Babelfish in 2021. Until then, you can use Babelfish in the Amazon Aurora preview to see how it works.
Figure 6: How BabelFish works
AWS Glue Elastic Views is a new feature for AWS Glue that allows you to easily build materialized Views that merge and replicate data across multiple data stores without having to write custom code. Glue Elastic Views uses AWS’s open source project Partiql, which provides SQL-compatible unified query access across multiple data stores containing structured, semi-structured, and nested data. Partiql separates the syntax and semantics of a query from the underlying data source and data format. It allows users to interact with data with or without a general schema. In addition, in the new release of Amazon Dynamodb, we also provide support for querying, inserting, updating, and deleting table data using PartiQL.
Amazon Neptune ML is a new feature of Neptune’s graph database that uses graphical neural networks (GNNS) for simple, fast and more accurate predictions. Using Neptune ML can improve the accuracy of most graphical predictions by more than 50% compared to using non-graphical methods. Deep Graph Library, an open source Library provided by AWS, is used behind Amazon Neptune ML. It is an open source project led by AWS Shanghai Artificial Intelligence Research Institute.
Apache Airflow is an open source project that builds batch data workflows to manage increasingly complex data management tools, scripting, and analysis tools. Amazon Managed Workflows for Apache Airflows (MWAA) is a Managed Apache Airflow service that allows you to choreograph data processing Workflows without having to manage, configure and extend your Airflow environment. Its execution is managed through logging and monitoring capabilities supported by AWS. You can run existing Airflow workflows on Amazon MWAA and programmatically interact with your environment using the AWS console, API, and command-line interface (CLI), while spending less time managing your infrastructure and Airflow environment.
Finally, let’s look at machine learning and edge computing.
Application scenarios such as image classification and natural language processing in deep learning are increasingly demanding in computational power and datasets, such as the 340 million parameters used in the Bert released in 2018. Now, more advanced models such as T5, GPT-3, Turing-NLG, and Megatron have set new accuracy records, but require tens to hundreds of billions of parameters. Manual implementation of a data and model parallelism strategy for distributed training can require several weeks of experimentation, slowing down the ability to iteratively deploy new releases into production. In the Re :Invent 2020 release, AWS offers a new hosted distributed training in Amazon Sagemaker, providing a faster and easier way to train large models and datasets. Sagemaker distributed training uses partitioning algorithm, which can automatically split model and training data sets between AWS GPU instances, and realizes data parallelism and model parallelism. Distributed training for data parallelism is used relatively often, with AllReduce being a key operation and a variety of open source implementations, such as Horovod. In Sagemaker Distributed training, Sagemaker Distributed Data Parallel (SDP) hosting has implemented optimized AllReduce; At the same time, AWS network infrastructure and Amazon EC2 instance topology are fully utilized to perform optimized node-to-node communication, which greatly improves the performance of distributed training. PyTorch-SDP is 41%, 52%, and 13% faster than PyTorch-DDP when distributed training Bert with 2, 4, and 8 nodes clustered on the P3DN.24xlarge instance. PyTorch-SDP was 4%, 19% and 15% faster than PyTorch-DDP when distributed training was performed on MASKRCNN. The specific results are shown in the following table, and the training code is located in Amazon-Sagemaker-examples.
Figure 7: Sagemaker Data Parallelism vs. PyTorch DDP
In addition, a growing number of applications, such as industrial automation and autonomous driving, require machine learning models that run on edge devices to make predictions in real time as new data becomes available. The newly released Amazon Sagemaker Edge Manager helps developers optimize, secure, monitor, and maintain machine learning models deployed on Edge device clusters. After the models are deployed to the edge devices, you still need to manage and monitor the models to ensure that they continue to run with high precision. As the accuracy of the model deteriorates over time, the developer can retrain the model to continuously improve its quality. Sagemaker Edge Manager uses Amazon Sagemaker Neo to optimize for different hardware platforms. Behind it is the open source Apache TVM compiler and the open source NEo-AI runtime, which currently supports devices based on chips made by Ambarella, ARM, Intel, Nvidia, NXP, Qualcomm, TI and Xilinx.
AWS extended AWS IoT to edge devices with the launch of AWS IoT Greengrass in 2016. AWS Greengrass has two main parts: IoT Edge Runtime and Cloud Services. Together, they are working to provide native computing, messaging, and data management capabilities for IoT devices. You can use Greengrass to run Lambda functions locally on IoT devices. Enable ML inference; Sending messages between devices; Deploy, run and manage Docker containers, etc. At Re :Invent 2020, AWS launched IoT Greengrass 2.0 with new developer features, including an improved command-line interface and an open-source edge runtime for adding and removing pre-built components to IoT Greengrass 2.0.
This is an overview of the major open source related releases in Re :Invent 2020. We believe open source brings benefits to every developer and are committed to bringing the value of open source to our customers and to bringing cloud operational excellence to the open source community. Of course, a healthy community is also important for developers to use open source. As a result, we are constantly looking to increase our support for the open source community in a variety of ways and have joined a number of open source foundations. As part of the foundation, we can participate in the design and discussion of the strategic direction of open source projects to help ensure that these projects continue to evolve in accordance with our clients’ expectations of us. And over the past few years, we’ve been speaking at various open source conferences to share our open source knowledge and experience.
Figure 8: Some of the open source foundations that AWS has joined
AWS provides a vast arena for developers and the community. The AWS Heroes project builds a dynamic, worldwide group of AWS experts whose passion for knowledge sharing is making a real impact within the community. There are already five AWS Hero operators in mainland China. The AWS Community Builders program provides technical resources, mentoring, and networking opportunities for AWS enthusiasts and emerging thought leaders who are interested in sharing knowledge and interacting with the technology Community. We also hope that more developers will join the AWS China developer community for broader and deeper communication.
