Author: Jade chase & rope

Youku and Huawei have maintained a good strategic cooperative relationship for a long time, aiming to bring quality video and audio entertainment experience to consumers. The circulation feature of Hongmeng operating system brings a new way of playing for multi-screen interaction. This paper takes the technical reserve of Youku Playback center as the starting point and combines the mirror and circulation characteristics of Hongmeng system to introduce the practice of ordinary circulation, free perspective and Zoom and other core capabilities in Hongmeng in detail.

background

Hongmeng Distributed experience

Huawei defines distributed experience as continuous experience and collaborative experience.

Continuous experience

When a user initiates an operation on one device and switches to another device to continue the operation, the user can immediately continue the current operation on the new device. Continuous experience includes task continuity and audio and video continuity.

Collaborative experience

Software and hardware capabilities across multiple devices work together to provide a more efficient and immersive experience for the user as a whole than on a single device. Collaborative experience includes software collaboration and hardware collaboration.

For Youku, we have provided users with the following multi-screen interactive functions.

• The user watches the video on their phone to a certain point in time, and then switches to the tablet to continue watching the video from where it left off
• Users open a video on their phone or tablet and use the screen projection function to project the video onto a smart screen or TV box to continue watching on the larger screen
• When a user encounters functions that are inconvenient to be completed by the big-screen client, such as login or payment, he can use his mobile phone to directly scan the TWO-DIMENSIONAL code on the big-screen for login or payment.

The above functions are directly realized by youku client using the existing “Huawei Touch transmission”, “Huawei HiPlay” and other functions. We hope to make use of the features provided by The Hongmeng OS to achieve some “new functions” that the current Youku client does not have, and develop a number of “pure Hongmeng” proprietary features, and then integrate with Android Youku. Hongmeng code and Android code linkage, together for consumers to provide a variety of cool video and audio entertainment functions.

Youku client spans Android, iOS, iPad, OTT and other hardware and software platforms. It can interact with each other through screen casting and other functions, naturally providing multi-screen interaction scenes.

HarmonyOS uses a variety of distributed technologies to enable application development regardless of device morphology. This allows developers to focus on upper-level business logic, making it easier and more efficient to develop applications.

In The Hongmun version, we built on the distributed bus capabilities provided by HarmonyOS and combined them with the existing screen casting capabilities of Youku to develop the new HarmonyOS Multi-screen Interaction FA, which provides multi-screen interaction between multiple Hongmun devices.

This FA is also written by 100% using hongmeng API. Users can transfer the video that is currently playing on the mobile phone to other Hongmeng devices (Hongmeng Smart screen, tablet, etc.) through the “Video Transfer” button on the playback page of Youku Hongmeng, and control the playback on the large-screen devices by using the mobile phone as a remote control.

Not only can you control the volume, fast forward and rewind, playback speed, clarity, and episodes on the large screen, but you can also rotate the Angle of the free-viewing video on the large screen.

The main difference between HarmonyOS-based multi-screen interaction and traditional screen casting is that we use HarmonyOS ‘Device/Service Discovery mechanism to search for peer devices and HarmonyOS’ Establish connections feature to establish two-way communication between devices.

Due to the highly optimized device/service discovery and Connection establishment features of HarmonyOSOS, our own experience was that HarmonyOS Multi-Screen Interaction was faster in device discovery, faster in connection establishment, and more stable than traditional DLNA or Miracast imaging.

In addition, we can use this connection for high-speed data transfer, directly from the control side streaming video to the large screen.

Below, let’s take a look at how HarmonyOS ‘distributed bus capabilities combine with Youku’s existing screen casting capabilities.

Screen casting service Overview

Screen projection technology has been applied more and more in our daily life and work. Wireless screen display and extension functions such as mobile phones, tablets and personal computers have also brought more convenient and efficient mode changes to family and enterprise meetings, product launches and remote training. There are also numerous product solutions in the market, such as Apple’s Airplay wireless projection solution, WiFi Alliance’s Miracast wireless projection solution, Google’s ChromeCast wireless projection solution, Intel’s WIDI wireless projection solution, etc. All have promoted the upgrading and evolution of multi-screen interactive products in home and office to varying degrees.

In fact, the improvement of wireless screen projection technology and network environment has marked the advent of efficient intelligent era, people can use mobile phones to surf the Internet, watch TV, entertainment, office, smart TV in addition to watching TV, more tend to the concept of smart screen, display screen. People want to be able to browse information, shop, play games and so on. People want to enjoy the immersion of watching TV on a big screen. The future development trend is also obvious, the huge potential demand of the market is the biggest driving force of screen projection technology, will present a “prosperous scene”.

As one of the domestic Internet video giants, Youku has accumulated rich technology and product experience in screen casting business, and has achieved very rapid development. In addition, Youku’s unique playback capabilities, such as free viewing Angle, ZOOM, AI super sub, 4K frame sharing, also provide a strong expansion space for multi-screen interactive scenes. On the whole, youku has laid a solid foundation for its cooperation with Huawei, both in terms of its screen casting business and broadcasting capability.

The scene is in flux

HarmonyOS is the next generation of smart terminal operating system that provides a unified language for smart, interconnected, and collaborative devices. HarmonyOS delivers simple, smooth, continuous, secure, and reliable full-scene interaction. Since its inception, HarmonyOS ‘mission has been to “connect Everything.” HarmonyOS uses a suite of code and core distributed technologies to meet the needs of a wide range of hardware across different platforms, including phones, tablets, computers, cars and smart wearables. It can be said that circulation is one of the core characteristics of Hongmeng ecology.

With the arrival of the intelligent era, circulation is everywhere, whether it is the two-way control of the vehicle, or the touch of the smart home; Whether it’s the touch of a document in the office, or the hyperlink between a smartphone and a smart screen, the flow can be achieved wherever there is a screen. On the other hand, the 5G era has redefined the way of entertainment and culture for the masses. Larger screen, higher bit rate and more intelligent circulation have become the core demands. In this context, Hongmeng and Youku naturally come together.

Youku screen projection capability map

Youku screen casting capability not only covers traditional LAN protocols, such as DLNA and AirPlay, but also develops a new cloud screen casting protocol, which completely breaks the restriction of LAN multicast and improves the success rate of device discovery. In addition, in order to add new gameplay, the latest magic screen devices come with NFC stickers, enabling a touch throw function, introducing a new gameplay. The next detailed introduction of youku screen casting ability map.

Ability to map

Wireless screen projection has three basic elements: content, device and protocol. The content can be non-real-time data files, such as movies, pictures, and music, or real-time data streams, such as screen images and live streams. Devices include the content provider initiating the screen projection (referred to as “sender”) and the content presenter receiving the screen projection (referred to as “receiver”). The common sender includes mobile devices such as mobile phones and tablets, which are responsible for obtaining media resources stored remotely or locally and transmitting them to the receiver. The receiver is responsible for responding to the content and broadcast control instructions from the sender and completing the subsequent content presentation. Protocols provide interaction protocols between sender and receiver, covering core links such as device discovery, data transmission, and playback control. Representative protocols include SSDP and SOAP protocols in LAN screen casting scenarios, MTOP and ACCS protocols in cloud screen casting scenarios, etc.

Multi-screen interactive services have high complexity and long interactive links, which span two business lines of large and small screens and involve many business parties including advertising, members, operators, broadcast, hardware, media assets and so on.

After long-term polishing with a large number of demands, Youku finally forms the following multi-screen interactive layout, which can be divided into screen casting protocol layer, screen casting and broadcast control layer, business adaptation layer and other core components:

Screen casting protocol layer

1. LAN screen casting. Including DLNA and AirPlay, with strong versatility, but limited by multicast capacity, the problem of device discovery often occurs.

2. Cloud screen casting. Cloud screen casting is a self-developed new protocol to solve the problem of device discovery by LAN protocol, which can effectively improve the success rate of device discovery. However, due to self-operated large-screen applications, its versatility is greatly limited.

3. NFC screen casting. As one of the new features of magic screen new hardware, NFC screen casting with attached stickers can shorten the operation link of screen casting on the mobile terminal and realize the “one touch casting” of Youku video.

Screen casting control layer

1. DRM management. Responsible for the management and calibration of screen projection equipment to ensure the correct release of copyright video;
2. Format processing. Adapt and compatible with different large-screen devices to ensure the playback experience after screen casting;
3. Clarity management. Be responsible for adaptive and abnormal degradation of multi-channel clarity, and take into account business scenes such as advertising;
4. Auxiliary scheduling. Based on dedicated domain names, the system can manage and schedule dynamic flow information of different regions, carriers, and devices.

Service adaptation layer

1. Screen ads. Including inside throw inside, inside throw outside, outside throw inside three combinations. In addition, the large and small screen advertising delivery system is also under development;
2. Point straight short fusion. The adaptation module of the three scenes of vod, live broadcast and short video is sunk to the internal maintenance of screen casting, which is of great significance to the unification of monitoring links and the external transmission of screen casting ability in the later stage.

Preliminary practice of Hongmeng system

General circulation

“Hongmeng Flow” refers to the multi-screen interaction mode using the characteristics of Hongmeng operating system. The biggest difference from the traditional screen projection protocol is its dependence on Hongmeng operating system. Specifically, hongmeng operating system not only provides “device/service discovery mechanism” as the protocol for device discovery, but also provides “device connection capability” to support bidirectional communication between devices.

Similar to the traditional screen casting protocol, the cloud flow includes common flow based on URL and flow transfer flow. Common flow refers to the multi-screen interaction mode based on hongmeng’s transmission capability. Different from DLNA and other general protocols, Hongmeng provides core capabilities including device discovery, data transmission, application query, file sharing, etc. It can not only provide more stable connection than the original protocol, but also according to business demands. The self-developed APK (such as Kumeow) is transferred from the small screen to the large screen and aroused to achieve a series of customized operations.

The general flow is introduced first, and its timing diagram is shown below.

As shown in the figure above, the implementation of some of the core processes is briefly described.

Hongmeng equipment found

The discovery protocol of Hongmeng is implemented in the search method of DlnaDevs, which is completely consistent with the process of DLNA and cloud screen casting:

Search if (HarmonyCastMgr.haveinst ()) {harmonycastMgr.getinst ().searchDevs(); Search if (HarmonycastMgr.getinst ().hasmirr ()){harmonycastMgr.getinst ().searchHarmonyMirrorDevs(); }}Copy the code

Let’s look at the implementation details of the Hongmeng discovery process:

Public Boolean searchDevs() {Boolean ret = false; if (isHarmonyEnable()) { Intent intent = new Intent(); ComponentName componentName = new ComponentName(PackageContant.PACKAGE_YOUKU, "com.youku.feature.MiddlewareAbility"); intent.setComponent(componentName); intent.putExtra(AbilityUtils.PARAM_KEY_INSTALL_ON_DEMAND, true); intent.setAction("action.videoplayer.getdevicelist"); try { mActivity = YoukuContext.getTopActivity(); ret = AbilityUtils.connectAbility(mActivity, intent, mServiceConnection); } catch (RuntimeException e) { e.printStackTrace(); ret = false; } } return ret; }Copy the code

The Hongmun discovery protocol relies on THE FA channel through AIDL, where mServiceConnection internally parses the IBinder object, maps the Device type defined for multi-screen interaction, and calls back the update interface of the Device list. The core implementation is as follows:

public ServiceConnection mServiceConnection = new ServiceConnection() { @Override public void onServiceConnected(ComponentName name, IBinder service) { LogEx.d(tag(), "onServiceConnected"); HarmonyRCS rcs = new HarmonyRCS(service); String[] devices = null; try { devices = rcs.getDeviceList(); } catch (RemoteException e) { e.printStackTrace(); List<String> deviceIdList = new ArrayList<>(); List<String> deviceNameList = new ArrayList<>(); for (int i = 0; i < devices.length; i++) { if ((i & 1) == 0) { deviceIdList.add(devices[i]); } else { deviceNameList.add(devices[i]); If (devicenamelist.size () >= deviceidList.size ()) {mHarmonydevs.clear (); for (int i = 0; i < deviceIdList.size(); i++) { String deviceId = deviceIdList.get(i); String deviceName = deviceNameList.get(i); if (StringUtils.isNotBlank(deviceId) && StringUtils.isNotBlank(deviceName)) { Client client = new Client(); . mHarmonyDevs.put(deviceId, client); }} // Check whether the large-screen device is online. If not, Stop for screen if (DlnaApiBu. API (). Proj (). The stat () = = DlnaPublic. DlnaProjStat. PLAYING) {Client currentDev = DlnaApiBu.api().proj().req().mDev; if (currentDev ! = null && currentDev.isHarmonyDev() && ! mHarmonyDevs.containsKey(currentDev.getDeviceUuid())) { stopProjEx(); }} // The caller gets the device list if (null! = mRcsCallback) { mRcsCallback.onHarmonyDevsChanged(); } } try { if (mActivity ! = null) { AbilityUtils.disconnectAbility(mActivity, mServiceConnection); } } catch (Exception pE) { pE.printStackTrace(); } } @Override public void onServiceDisconnected(ComponentName name) { LogEx.d(tag(), "onServiceConnected"); }};Copy the code

Select the device and start FA

Similar to the cloud projection self-development protocol, The Hong Kong Normal Flow relies on a separate discovery process. Similar to other protocols, HarmonyCastTrunkBiz was created as the Hong Kong Protocol management class and implemented the following basic interfaces:

// Start screen cast void start(); Void stop(); Void play(); Void pause(); Void seek(int prog); Void setVolume(int volume);Copy the code

Since Hongmeng FA provides customized remote control interface and two-way data transmission channel, all tasks of sending instructions are taken over by Hongmeng FA, and the interface responsible for sending instructions only synchronized the playing state of the small screen. When the user selects the Hongmeng device, the start method is triggered and the Hongmeng protocol entry is initiated. StartHarmony from HarmonyCastMgr is responsible for data interaction with The Hongmeng FA:

HarmonyCastData = new HarmonyCastData(); castData.mDev = req.mDev; castData.mUrl = req.mUrl; castData.metaData = DlnaMetadata.getInst().getMetadataWithReq(req); castData.isFromNowbar = isFromNowbar; ParameterBean = new HarmonyParameterBean(); HarmonyParameterBean = new HarmonyParameterBean(); parameterBean.deviceUuid = req.mDev.getDeviceUuid(); parameterBean.mShowTitle = req.mShowTitle; parameterBean.projSource = ""; parameterBean.isYoukuApp = req.mDev.isYoukuApp; String parameterData = JSON.toJSONString(parameterBean); String touchuanData = json.toJsonString (castData); HarmonyPaBean paBean = new HarmonyPaBean(); paBean.touchuanData = touchuanData; paBean.parameterData = parameterData; final String harmonyJson = JSON.toJSONString(paBean); Activity activity = YoukuContext.getTopActivity(); Intent jIntent = new Intent(); ComponentName component = new ComponentName(PackageContant.PACKAGE_YOUKU, "com.youku.feature.MainAbility"); jIntent.putExtra(AbilityUtils.PARAM_KEY_INSTALL_ON_DEMAND, true); jIntent.putExtra("harmonyJson", harmonyJson); jIntent.setComponent(component); Try {// Check whether the remote device is online before each screen casting to avoid screen casting failure caused by a large screen offline Boolean remoteValid = true; if (HarmonyCastMgr.haveInst()) { remoteValid = HarmonyCastMgr.getInst().searchDevs(); } if (remoteValid) { AbilityUtils.startAbility(activity, jIntent); } else { stopProjEx(); } } catch (Exception e) { e.printStackTrace(); }Copy the code

Free view

Developed by Ali Entertainment Moku Lab, Free viewing Angle is a kind of playback technology that can slide video and switch viewing Angle freely. Free view video is a further development of multi-view live broadcasting, bringing users more initiative. Users can rotate the perspective while watching the game, and zoom in on the partial picture to closely watch the dunk action of their idol players, which can realize the real movie viewing experience comparable to trans-dimensional.

Here is the free-angle effect of this is Hip-hop:

Video please see the original text: Youku Hongmeng Development practice | multi-screen interactive development practice

Different from ordinary video, the bit rate of free view video is generally high. The street dance recording the scene, for example, installed a total of more than 40 sets of special camera, forming the array collection picture of all-round record every moment, and realize the coordinated control of millisecond synchronization and remote, collected images after focus correction and 3 d reconstruction, and based on the flight reservation to texture and depth, parameters, such as information, Finally, the multi-angle scattered pictures are splicing into a 6K, 8K high definition video file.

Therefore, free view video not only improves the user’s perception, but also puts forward very high requirements on the network environment and decoding performance of the player. The original free view implementation scheme is based on the flow feature. The small screen transmits the M3U8 address or broadcast control command of the free view video to the large screen through the above flow capability, and the large screen completes the subsequent processes such as data download, decoding and rendering. However, considering the hardware performance of Huawei smart screen is not enough to support this scale of computing, the traditional flow scheme had to be stopped.

To this end, the development team compared all existing screen casting protocols on the Hongmeng system and came to the following conclusions:

Hongmeng screen projection protocol	advantages	disadvantages
DLNA	Standard mature protocol, strong versatility	Native scalability is weak
Cloud cast screen	Independent of LAN, strong scalability	Poor versatility
HongMeng agreement	The system has high adaptability and strong expansion capability	Rely on the Hongmeng system
Cast + agreement	The system is highly adaptable and compatible to meet the needs of stream transmission	Rely on Huawei Ecology

After internal evaluation and discussion with Huawei r&d team, the development team finally decided to abandon the traditional URL-based screen projection method and adopt huawei Cast+ based mirror screen projection scheme. Specifically, it is to normally play and control free perspective video on mobile phone and mirror the player view to TV through Cast+.

This section describes the implementation details of Huawei Cast+ protocol. As shown in the following figure, Huawei Cast+ Kit is a mobile phone-centric multi-screen collaborative component developed by Huawei, enabling services to achieve fast, stable and low-latency image transmission among multiple devices.

Based on Huawei Cast+ protocol, the screen casting link of free-view scene is basically the same as that of ordinary screen casting protocol, and the overlapped part includes device search, device selection and connection construction. The difference is that the last step of free-view video does not put the playing address on the large screen, but creates a virtual screen and displays it. The time and way of creating virtual screen is the most important in practice.

The essence of the free perspective transfer scheme can be summarized into two aspects: first, the remote control page is used as the main screen to open the two-way data channel between it and the playback page to realize the playback control and state synchronization; Second, start playing the relevant view on the virtual screen as a mirror and transfer the content to the larger screen.

The plan for the home screen was pretty much settled, but when it came to the virtual screen, it got stuck. At first, the virtual screen rendering object that can be thought of is naturally a full-screen page. However, after trying it, I found that directly rendering the full-screen page to the virtual screen would have the following two problems, and the attached analysis report is as follows:

Problem 1: The remote control page on the home screen is first mirrored to the larger screen.

• Cause: Cast+ will capture the content of the virtual screen when the device is in PLAYING state after the connection is successfully established. If the virtual screen has not been loaded, the content of the main screen will be captured as the bottom pocket. After testing, the time difference was about 100ms, during which the playback page could not be initialized, so there would be a problem of occupying the content of the main screen.

Fault 2: After entering the playback page, there is a logic to switch between vertical and horizontal screens. As a result, the screen rotates once after mirroring.

• Cause: It belongs to the adaptive logic of the playing page. When entering the screen, it is portrait. After mirroring, the landscape switching logic will be triggered, resulting in screen switching on the large-screen side.

After a comprehensive internal evaluation, we decided to adopt a more lightweight approach to virtual screens. We find that the player Dialog contains only SurfaceView and Loading state and other contents. If the Dialog level is only mirrored, more player processing logic is added, but the on-screen speed and response efficiency are ensured. The adjusted scheme structure is shown in the figure below.

The scheme has the following advantages:

1. Optimized the life cycle of the player, every time when switching between free view flow and small screen playback, only switch SurfaceView, not rebuild the player, at this time the player is only in the STOPPED state, after cutting to small screen playback, skip initialization and play address request process, to ensure fast playback;
2. The time to start Presentation will be advanced, and the page will be started and the player will continue to play when onDisplayAdded is received, greatly shortening the on-screen time.

To sum up, the final effect is shown in the following video.

Video please see the original text: Youku Hongmeng Development practice | multi-screen interactive development practice

Online performance

Netizens have also shown strong interest in the characteristics of hongmeng system and the performance of Youku Hongmeng version. We have selected some online experience videos and listed them as follows:

• “Youku HongMeng edition experience – turn self-knowledge on: www.bilibili.com/video/BV1A5…
• The HongMeng youku screen Love the love “: www.bilibili.com/s/video/BV1…

Subsequently, after two versions of continuous iteration, youku Hongmeng’s stability and experience has been greatly improved. On the other hand, with the self-improvement of hongmeng system, youku Hongmeng version of the second demand version has begun to schedule, I believe that under the strong combination of both sides, will continue to bring surprises to the market and users.

conclusion

In youku HongMeng version screen interaction more special development process, all the participants in HongMeng interface and debugging environment is not perfect time to step in and overcome the difficulties, and continuously explore trial and error, in a very short period of time to master the HongMeng development technology stack, not only successfully completed the development task, big and small, still feedback to huawei technologies team many legacy problems and Suggestions for improvement. This is another in-depth cooperation between the two sides and a very successful practice.

For Huawei, Youku Hongmeng not only brings a pioneering new form of video interaction to Hongmeng Ecology, so that new users shout great; And perfectly adapted to the flow characteristics of the Hongmeng system, paving the way for the subsequent richer new gameplay. For Youku, the practice of multi-screen interaction on Hongmeng greatly expands the existing multi-screen interaction capability map and accumulates important experience for the new era of Hongmeng.

Follow us every week for 3 mobile technology practices & dry goods for you to think about!