Abstract: Huawei Cloud FusionInsight MRS a new-generation data lake to make the use of big data faster, easier, more stable, and more efficient! Let the value of data in front of you!
On October 30, the second Data Analysis Technology and Application Summit forum was held in Shenzhen, with the theme of “Win-win together · Create the Future with several Numbers”. At the meeting, Wang Ning, senior marketing manager of Huawei Cloud Big Data, delivered a keynote speech “Huawei Cloud FusionInsight MRS spanning the technical rift valley, helping customers realize one enterprise, one lake, one city and one lake”!
The focus of big data technology innovation has shifted to LakeHouse, and the focus of enterprise innovation has shifted to the integration of lake and warehouse
“With the growing maturity of big data technology and the large-scale commercialization of 5G, AI and IoT technologies, government and enterprise customers need to be driven by business demands and technological innovation to realize digital transformation. It is very important for enterprises to choose a competitive digital base to build a technology leading big data platform. In recent years, Huawei cloud Big data keeps pace with the world, keeps innovating, and is committed to building a technologically leading digital base, so that the use of big data is faster, easier, more stable and more economical! Leave the complexity to yourself, leave the simplicity to partners, help customers digital transformation success!” “Wang Ning shared. So digital base is so critical, big data as the main bearer technology, how will it develop in the future?
After 2011 and the evolution of big data, as the process of informatization enterprise began to use big data technology to build data such as lake, meet the diversity of data storage, data lake, while suitable for storing data, but lacks some support the key ability of business: does not support real-time incremental updates, for example, does not support transactions, unable to real-time analysis, etc. It is predicted that the global data volume will grow rapidly from 33ZB in 2018 to 180ZB by 2025. In order to cope with the exponential growth of data assets, customers need to adopt large-scale, efficient, one-stop, multi-scenario big data processing system more and more urgently. By 2020, the focus of big data technology innovation trend has shifted to LakeHouse, and the focus of enterprise innovation has shifted to the integration of lake and warehouse.
The data lake develops to LakeHouse. Based on the data lake, the data structure and data management capability similar to the data warehouse are realized, and real-time update, BI interactive analysis and mixed load are supported. Data lake and data warehouse are developing towards integrated analysis architecture. Hadoop and MPPDB have data sharing and cross-database analysis capabilities, support interconnection, computing push down, and collaborative computing, so that big data and storage are interconnected and collaborative computing, taking into account the past and future! Huawei cloud big data technology synchronizes with the world, actively embraces open source, and draws on the world’s top practical experience in big data. Being closed leaves people behind, and other mainstream vendors have embraced open source in recent years. Open source and open technology of big data are still flourishing. In the past, the “troika” of big data can turn around the pond of big data. Now, big data technology has developed into an ocean, and the community has 100+ open source projects. Now big data is not limited to the Hadoop ecosystem, but a collection of various mainstream data processing technologies, supported by rich components in different scenarios.
Huawei Cloud creates a cloud yuansheng data lake with leading technology, realizing a lake for each enterprise and a lake for each city
As a data lake leading in technological innovation and market share, Huawei Cloud FusionInsight MRS has three core capabilities:
1) Let government and enterprise customers continue to evolve under a large, fast, harmonious and stable cloud native data lake architecture
Large: The capacity expansion of a single traditional big data cluster is limited, and it is difficult to break through when the cluster is expanded to 2000 nodes. There are problems such as complex operation and maintenance of segmented cluster, high cost, low resource utilization, and failure to guarantee key tasks. Huawei FusionInsight MRS breaks the scalability bottleneck through large clusters. It supports a large-scale cluster with a maximum of 20,000 + nodes. In addition, the cluster federation can be expanded indefinitely, enabling government and enterprise customers to continuously evolve in a single architecture. In terms of BI enhancement, it can realize the millisecond real-time analysis of thermal data and efficiently support BI. Enhanced Spark to support JDBC multi-instance and increase the number of BI report connections!
Large-scale features have been practiced within Huawei. Huawei GROUP IT builds a OneData big data cluster through FusionInsight to expand the big data platform in large-scale scenarios. The size of the OneData cluster has reached 10,000 + nodes. At the same time, the unified data management service is realized. In the UniDB product of pudianhai, 50+ physically dispersed computing clusters (Hadoop+MPP) are logically unified to form a unified architecture of near lake and warehouse, supporting nearly 60PB data analysis requirements of thousands of enterprise tenants. Huawei group’s IT OneData large cluster has undergone two rolling upgrades with 0 service interruption and has been running steadily for six years. The largest single cluster of Huawei cloud FusionInsight supports 20,000 super-large clusters, making the customer’s service system as stable as a rock and having no worries in 10 years.
Fast: the increasing of data is bound to bring the problem of lower analysis efficiency. Huawei Cloud Big data breaks the performance bottleneck, directly faces the customer’s business to build subject data, and analyzes the short link, the faster the more use, no need to wait! Traditional big data has the problems of “slow, difficult and expensive”, such as long data links and post-event reports. FusionInsight MRS can update data incremental in real time and implement real-time OLAP at millisecond level, so that big data analysis does not need to wait!
In a financial row, 100+ nodes are clustered in the row, and the total data volume is 1PB. 100,000 tables are updated every day. Pb-level data is synchronized in real time through FusionInsight MRS, and the storage time of 100+ nodes data is reduced from 12 hours to less than 1 hour.
It solves the problems of big data scalability and high performance. If government and enterprise users want to deeply release the value of data, they still need cross-source and cross-domain fusion analysis.
Melt: eliminate data island, five fusion, cross-source cross-domain cross-engine fusion analysis, so that data analysis is more and more simple, eliminate data island, without redundancy! Traditional big data analysis is faced with problems such as multiple types, scattered distribution and difficult coordination. FusionInsight MRS uses The HetuEngine to achieve the five integration, unified SQL interface, simplified usage, universal BI, make the use of big data more and more simple!
One row builds financial big data based on FusionInsight MRS. HetuEngine uses unified SQL interface to solve problems such as data dispersion, multiple components, and multiple languages, reduce technology development threshold, cross-source, cross-domain and cross-engine fusion analysis, data relocation is avoided, and overall TCO is reduced.
Huawei cloud FusionInsight MRS is not only faster and simpler to use, but also protects customers’ investment. It also cares about the stability and sustainable development of government and enterprise customers when using big data. No new reconstruction is required!
Stability: the growth of data is infinite, small clusters will always grow into large clusters, a technology leading, smooth upgrade, sustainable evolution of the base is very important. FusionInsight MRS can ensure the continuity of one enterprise and one lake, and ensure rolling online upgrade. Services are always online without dismantling clusters or transferring applications. FusionInsight MRS completely solves problems such as multiple clusters, low efficiency, difficult management, and difficult upgrade of traditional big data.
A carrier uses FusionInsight MRS to build a big data platform to cope with 5G data surge. Two rolling upgrades have realized the smooth evolution of the big data platform, achieving no service interruption, no upgrade awareness, and continuous online user experience. The size of a single cluster has expanded to 1500+ nodes. It supports 200+ big data application services of various kinds of government affairs and people’s livelihood, covering more than 130 million users, and makes customers’ business never stop, service always online, and technology always up-to-date!
2) Real-time data lake
Real-time data lake, let the data into the lake in real time increment, T + 0 real-time analysis; Realize real-time multi-dimensional analysis of source data, shorten the analysis link, improve the analysis efficiency, towards real-time data lake, so that the value of data is close at hand!
3) Cloud native data lake
Huawei cloud FusionInsight MRS has cloud native features such as unified metadata and separation of storage and computing. Through Data Lake Catalog, it provides unified metadata service for super scale analysis engine, making Data globally visible and accessible. In terms of data storage, OBS storage and calculation separation scheme is adopted to realize the capacity expansion of computing and storage on demand. Based on enterprise-level EC, the minimum copy is 1.2, the total TCO is reduced by 20%+, and the cost of data per bit is better! The interactive analysis engine is provided in the lake, which can seamlessly connect BI reports and self-service analysis, realizing second level usage without data relocation. The unified SQL interface in the lake lowers the threshold of technology development, simplifies the number of uses, and realizes the cloud native data lake of technology innovation.
The cloud native data lake of one enterprise and one lake has become a standard base for government and enterprise customers, providing one-stop support for efficient analysis of all scenarios
The data lake of the new generation of Huawei Cloud FusionInsight MRS enables the faster, easier, more stable, and more efficient use of big data. Let the value of data in front of you!
Huawei cloud FusionInsight MRS firmly adheres to the open route, adheres to feedback to the community, continues to invest, and keeps abreast with the world
Based on the strong innovation capability of Huawei cloud big data, Huawei Cloud FusionInsight has made outstanding achievements in the industry and has been recognized by many authoritative organizations for many years. In 2020, Huawei Cloud FusionInsight has been shortlisted among the top 50 Big Data Enterprises in China for four consecutive years, and won the China Information and Communication Big Data Industry Influence Award and China Big Data Platform Best Solution Award. Huawei Cloud leads the development of big data technology, understands the continuous development of customers’ business demands, and has sustained high intensity investment for more than 10 years. It owns more than 500 patents, and PMC and Committer accounts for nearly 50% of key areas. At the same time, Huawei cloud Big Data adheres to the strategy of platform + ecology, and serves global government and enterprise customers together with partners. Huawei cloud FusionInsight MRS firmly open route, peer with the world, continuous investment, do a good job in the digital world of the black land. At the same time, we insist on the good experience, continue to open to everyone in the Huawei cloud big data community, let the ocean can not block the determination to climb the peak of big data technology. Huawei cloud FusionInsight MRS keeps the “complexity” for itself and the “simplicity” for partners, to build a prosperous community ecology, and to unite 800+ISV to create a win-win situation and help the digital transformation of government and enterprises succeed!
_
_
Huawei Cloud FusionInsight unites 800+ partners to create a win-win situation and accelerate the digital transformation of government and enterprises
Recently, IDC will release 2020 China Big Data Management Platform manufacturer evaluation report, Huawei Cloud FusionInsight Intelligent data Lake with years of understanding of the industry business, adhere to technology innovation to lead the global big data development, I believe that will also produce a satisfactory answer.
Huawei cloud FusionInsight MRS has become a common choice for digital transformation of customers in 60+ countries and regions and 3000+ countries. It is widely used in government, operators, finance, energy, medical care, manufacturing, transportation, Internet and other industries to release massive data value, drive business growth with data, and help government and enterprise customers to realize “one enterprise, one river”. A city and a lake!”
Click follow to learn about the fresh technologies of Huawei Cloud