Article | InfoQ Xu Chuan
The original article was published by InfoQ- Top of the Front on November 11, 2017. This article has been modified with permission from the author. Read the original
Since 2012, Alibaba’s Annual Singles Day promotion has featured a large screen displaying transactions in real time in a variety of vivid displays. Real-time data large screen, which is characterized by large screen, large amount of data, large amount of information display, can be said to be the holy grail of data visualization.
(Figure: Lightning map of regional economy on Double 11 in 2017, showing real-time order data and logistics trunk lines)
Ali Cloud DataV team has participated in the digital screen construction of Double 11 media for 6 consecutive years. This year’s “Lightning Map of China Regional Economy” media touch screen has become one of the core focus of the Double 11 gala. Orders of hundreds of millions of buyers across the country are pouring into sellers along the real road on the screen, and the effect is shocking. The large touch screen allows users to drill down and query data, greatly improving the readability of data. On that day, more than 500 Chinese and foreign media took it as the data explanation platform for live broadcast, realizing the perfect integration of old and new media.
In this year’s Double 11 Data Big Screen project, a number of new data visualization technologies developed by DataV, such as “DataV Data Big screen Calculation and Rendering Load balancing”, “Global Logistics Trunk Line Visualization Model”, “DataV MScreen Multi-view Fusion”, “WebGL later visual effect enhancement”, were integrated and applied for the first time. In a single screen 3D rendering with super 12K resolution and concurrent access of 30+ real-time data sources, it can still ensure silky smoothness and stunning visual effects.
In the 2016 Double 11 media screen, the main vision is the 3d real city scene of Shenzhen. The color of the building represents the size of the order amount in the building, and the redder the color, the higher the order amount. The panel on the right shows the user’s crowd portrait and other information.
(Figure: Economic cloud picture of Shenzhen on November 11, 2016. The color of urban buildings represents the amount of orders in the building.)
As the behind-the-scenes hero of the Double 11 data big screen, DataV has tried almost all paths of technology precipitation after several large projects: open source code, productization, packaged solutions, platformization… Application scenarios have also expanded from Double 11 e-commerce operations to smart cities, smart transportation and many other fields. The course that they come all the way and ponder, be worth later to draw lessons from reference.
(Figure: Main vision of Hangzhou city brain, panoramic view of all aspects of the city’s traffic conditions and optimal scheduling)
The birth of DataV is accompanied by Alibaba Group’s exploration of data-driven operation.
In 2011, Taobao incubated its first data product — Data Cube, which aims to reduce the cost of understanding data for small and medium-sized sellers on the platform through vivid data charts, and then enable sellers to make use of store data in actual operation through the interaction of charts and other functions. The emergence of the data Cube popularized the value of digital operation both internally and externally. At that time, it was the current DataV team that was in charge of the visualization development of the data Cube. They soon undertook the technical upgrading and transformation of ali’s internal data big screen, and in this process, they also promoted the construction of Ali’s real-time data indicator system.
In 2012, Double 11 had its own media center for the first time — a training room turned into a temporary command center. The hardware for the big media screen was supported by two projectors and two laptops. At the beginning of the 2012 Double 11, the actual transaction amount exceeded the preset maximum value of the chart, and the GMV real-time line chart was off the charts as soon as it came out. Although there was no special sorting of indicators and insufficient experience in visual chart development at that time, the debut of the Double 11 large screen still brought people a huge visual impact and a real sense of data.
Until this year’s Double 11, DataV team has been responsible for the visual development of double 11 media data large screen for 6 years. The resolution of large screen has increased from 1024*768 to 12K, and the screen area has reached 400 square meters. The complexity of data display and real-time data link have changed dramatically.
While technology capabilities are being upgraded, the DataV team is shifting its role as an internal service provider to a visualization product and solution provider, exporting its real-time data visualization capabilities. At the same time, DataV is moving from pure data visualization to visual analysis of data.
At present, DataV is fully cooperating with the ET City Brain project of Aliyun to practice the visual analysis platform of city-level multidimensional data. The large screen of data command in the first phase has become a part of the daily work of Hangzhou traffic police, and a new round of technical solutions has been upgraded by combining the image recognition and other AI scenes of machine intelligence.
In addition, DataV is also working with many government companies to upgrade data command screens to help them quickly identify data motion in the early stages of digital operations management.
From open source to Productization
Early on, DataV precipitated the datav.js component library and made it open source, but later, DataV abandoned the open source project and made it productized. Why?
Open source is to assign the visualization technology can give more data show the scene, but in the process of promoting have met a lot of custom issue, developers a lot of problems in processing data and charts docking (such as D3 rich demo library for data developers or slightly higher a threshold, so open source and failed to reach a vision. At the time, the datav.js component library architecture was designed to support earlier versions of Internet Explorer, but demand for this became weaker and the library was gradually abandoned.
In this process, the DataV team observed that providing a visual component library is not the same as breaking down the full link of data kanban production. It lowers the bar for those who are directly exposed to or want to interact with the data, but beyond that, High-performance data visualization also involves styling, data wiring, component architecture, and other skills that are more complete when output is based on a product. So the DataV team started thinking about the transition path.
First of all, data visualization is a comprehensive reflection of business understanding, data understanding and visual expression in the system of large companies, although it is a bit front-end. It needs to be completed jointly by business, data, front-end and design. From a purely technical point of view, front-end components are only part of the data processing, how to docking is actually a more critical step in the visualization of business landing. Compared to the familiar ECharts development approach, DataV is more interested in connecting data, business, front-end, and design in a productised way to truly lower the barrier of entry.
This makes it easy for data developers to work on a professional chart front end, and for designers and businesses to make “moving” data kanban, dramatically reducing the engineering and communication costs of visualizing multiple stacks. Secondly, with a commercial product, the developer can more accurately understand the real needs of users through the communication and feedback of customers. While open source front-end components are widely used, developers themselves have no idea what scenarios others are using, making it difficult to feel pain points and appeals. And just from the front end, once the basic charts are developed, there is a bottleneck in direction.
DataV is currently positioned as a tool for visualizing analytics on the big screen.
In terms of functions, the most important thing is to provide drag-and-drop large-screen interface configuration tools, as well as a variety of templates. It is easy to achieve the effect of double-11 large-screen by making PPT templates. It also includes a variety of data connection capabilities, including common types of databases, API interfaces, etc.
(Image: DataV product interface)
In terms of features, DataV has made a big effort in the WebGIS space, integrating a variety of WebGIS capabilities, in addition to the regular, cool-looking basic chart components. D3. js, leaflet.js, mapbox.js, native canvas, SVG, traditional back end drawing map tiles, mapping and projection of various geographical data, as well as event response of multiple map layers on the map have been very well integrated. General BI or visualization tools are relatively easy to implement on basic charts, but the GIS part is much weaker. DataV can be easily integrated with multi-layer mapping technology and geo-related data integration through interface configuration.
(Image: DataV map component)
In 2016, Mir, a front-end development engineer of DataV, shared the Evolution of Data Big Screen Technology on Double 11, introducing the development process of DataV from an initial EXE program to large-scale WebGL application by taking double 11 as an example.
Speech finishing see: yq.aliyun.com/articles/73…
Datav. gl: Visual analysis solution for smart cities based on WebGL
After years of WebGL research and practice, DataV has launched the Datav.gl component to massively improve rendering performance and enable 3D digital city customization. From this point of view, Korea has achieved the world’s leading level.
DataV packages 3D visualization capabilities into an easy-to-use component. Users can import 3D model files and adjust data and styles through graphical menus just like editing other charts. In terms of implementation effects, Datav.gl supports rendering of buildings of 100,000 magnitude, urban pipeline lines and natural landscapes. Users can also customize the camera movement track to realize the free exploration of the city picture.
From the technical point of view, DATav.GL greatly improves the computational efficiency and engineering efficiency of 3D urban geographic visualization. Using classic GPGPU technology based on browser and the computing power of graphics card, Datav.gl supports smooth rendering of massive data in flow field, force mapping and other computationally-intensive visualization solutions. Compared with the traditional CPU-based Canvas force mapping scheme, datav. gl’s GPU acceleration force mapping scheme improves performance by more than 30 times.
At the same time, datav. gl supports urban data planning files (CAD and other formats), standard vector geographic data as input data, and can directly model 3d urban buildings through geographic data visualization mapping services, which is much more efficient than traditional designers using 3D software modeling often takes weeks.
(Figure: Datav.gl editing interface)
In the past two years, Uber has made great achievements in the field of THREE-DIMENSIONAL visualization of geographic information. The deck. GL front-end library released by Uber has become a benchmark in the industry. However, a single source visualization technology framework cannot provide big data visualization solutions for multiple scenarios across industries, and Deck. GL also has many limitations in meeting personalized needs and selecting front-end technology stack.
Uber Deck. gl is a visualization front-end library, which can solve the needs of data visualization development, while other links require users to find their own solutions. Datav. gl is a set of mass data high-performance visualization solutions, and users can complete data visualization business “one-stop”.
Using a car analogy, Uber Deck. gl is a car engine and Datav. gl is a complete car. Just buy the engine can’t go on the road, also need other parts to assemble.
The difference between datav.gl and deck.gl is as follows:
(1) From the perspective of function positioning, Datav. gl covers a wider range than Deck. gl. It not only provides the functions of a visualization library, but also provides a whole set of big data visualization solutions.
(2) In terms of target user groups, Deck. gl favors developers, while datav.gl favors enterprise and government users.
(3) From the comparison of user demand suitability, Deck. gl and React are deeply bound, and the basic official visual components are convenient to use in front-end development. However, some personalized requirements need in-depth analysis of Deck and React framework to develop, which is relatively high cost. Datav.gl offers a developer version where users can either use official components or build custom components based entirely on their own technology stack and development habits.
(4) Purely from the perspective of rendering efficiency, both Datav. gl and Deck. gl are based on WebGL and have similar basic performance. Datav. gl has advantages in some specific mass data visualization scenes, as mentioned above.
(5) From the perspective of visual material resource integration, Datav. gl provides fool components by accessing some resources such as live streaming media and 3D modeling. Deck. gl is relatively complex.
The future of the DataV
In terms of future direction, DataV focuses on visual ecology and technology exploration.
Although the big data screen can serve all walks of life and has great market potential, this market needs more ecological partners to join in and create different sparks, no matter in terms of each link of the big screen or the domain knowledge needed to finally enter the industry. For example, in THE ET City Brain project, a number of ecological forces and DataV joined forces to form a complete set of large screen solutions for the visual analysis of city-level data.
, of course, as a technical team, DataV is also very important platform of visual developers, they launched the developer edition, can carry on the secondary custom development, let developers follow certain development specification, you can put the homemade components into products, like use native components can be graphical configuration. As well as enabling developers to publish their own components to the Aliyun market and realize the realization of visualization capabilities — this is a continuation of the DataV team’s original intention to promote the development of data visualization and find a commercial outlet for visualization.
At the same time, DataV also subtly shares its technical ideas with external developers through technical documents, and provides real and accurate improvement directions for developers based on the massive usage data deposited by the platform. Not only do we want end users to feel the value of data visualization, but we also want to pass this value on to the millions of developers working on visualization.
DataV another direction is, will continue to strengthen the investment in the visual analysis research direction, in technology to optimize product framework supports data more powerful interactive function, based on the theory of visual strengthen the cooperation of colleges and universities and research institutions such as, to ensure its own data visualization direction have stronger lasting power support.
Introduction of guests interviewed:
Ning Lang, Head of DataV data visualization technology, head of commercialization of Data Visualization Group of Alibaba Front-end Committee.
Yong Ling is product Manager of DataV and head of Aliyun City Brain product.