The author | WuShuang

Planning | bag enquiry, Alex

Edit | Alex

Interactive whiteboard year-end inventory #005

Miro, an interactive whiteboard, recently raised $400m at a valuation of $17.5bn. The news has also brought a lot of attention to what was once a niche area: interactive whiteboards.

Why do we need interactive whiteboards?

In conference or teaching, we often need voice and video communication plus presentation materials. So in general, real-time audio and video plus a screen recording function is enough to cover mainstream needs.

But we observe the mainstream conference software or teaching software is not difficult to find a rule, they are equipped with a powerful interactive whiteboard system. So when do we need interactive whiteboards to assist in teaching or meeting, or what are the advantages of interactive whiteboards that are hard to replace compared to screen sharing?

  • Low bandwidth usage *

Screen recording is actually multiple video sources, and the essence of whiteboard synchronization is signaling synchronization. The upstream bandwidth required for screen recording is several hundred KB /s in most cases, and even MB /s in HD and dynamic cases. The signaling synchronization of whiteboard is basically tens of KB /s. There is less bandwidth required to maintain a long-connected heartbeat with no synchronization information.

Low bandwidth usage allows more bandwidth to be reserved for voice and video calls, which further improves overall stability.

  • Look and feel clear

The clarity of a recorded display depends on the resolution, which in turn depends on the quality of the network (and possibly the cost, since the high bit rate and cost of high-definition recording). Especially when the data text is relatively small and the network is unstable, the viewing experience of recording screen will become very bad.

The interactive whiteboard display data will generally do a transcoding preprocessing to ensure the clarity and accuracy of the data. Therefore, using whiteboards to present data in meetings is often more stable and clear.

  • interactive

The interactive whiteboard is real-time interaction based on signaling, which is closer to online games on the technology stack, so it naturally has a good interactive experience. Teachers and students can doodle, edit and other interactive operations in the whiteboard at the same time. Preschool education customers often make H5 courseware with game interest based on the signaling system of the interactive whiteboard. But screen sharing cannot realize online whiteboard interaction between teachers and students.

  • Mobile friendly

Because the power consumption of the recording code is reduced, the power consumption of the software is reduced, and the device is powered down more slowly, so the experience is better.

  • Good privacy

In addition, the use of whiteboard to display information, without authorization screen recording, better privacy protection.

What technologies are the interactive whiteboards built on

Through the above analysis, we have a detailed understanding of the advantages of the interactive whiteboard. So let’s talk about the key techniques to build a useful interactive whiteboard.

  • The overall structure of an interactive system

* Back-end system: As shown in the figure above, database + server + long connection constitute a complete back-end service. It is not difficult to see that we need a real-time database to complete the most critical real-time state synchronization in the interactive whiteboard.

* Front end systems: Once the state is synchronized in real time and consistent, we need to render a consistent view of the state. Then the Web is the most common choice across the Web.

Let’s take a look at real-time database services and cross-platform synchronization.

  • Real-time database service

What kind of real-time database is a service that meets the requirements of interactive whiteboard system? Simply put, the data layer should be reliable and efficient; The real-time network layer should be stable and low delay. The application layer must meet the specific requirements of whiteboard scenarios. Let’s talk about each part in detail.

  • Data layer difficulty: How to persist the site and quickly recover the site?

– 1. How to persist the site?

The system must have the ability to record the scene if it wants to join in or out and continue to synchronize and interact. We call this capability “Live data persistence.” Let’s look at the diagram together.

When multiple members interact in real time, we start a service that subscribes to the state in the interactive server’s memory and writes the state to the database. Because the database stores the scene, someone can read the state of the scene from the database after joining in the middle, and then connect to the interactive server to participate in the scene quickly.

– 2. How can I quickly recover the site?

Simply documenting and restoring the scene does not improve the interactive experience. With the increase of interaction time and content, the content to be read and the action of recovery become complicated.

We learned the idea of audio and video coding, and used the concepts of “full frame” and “incremental frame” in the signaling system, as shown in the figure below.

The system generates “full frames” every 300 actions, so that when exiting the room and re-entering, full frames and a small number of incremental frames are available to restore the scene. You don’t have to go through it all again. In a room that has been in use for a long time, this method can significantly improve the speed of entry.

  • Network layer: protocol optimization

The interactive whiteboard is usually used with real-time audio and video. In this case, the UDP-based RTC occupies more network resources, and even the whiteboard system fails to send any messages. However, whiteboard can do some protocol layer optimization in Native terminal, and also use UDP protocol to transmit signaling, and ensure the arrival of messages in the protocol upper layer. As for the overall optimization strategy, we can directly refer to the optimization ideas of QUIC. Due to the limited space, we will not discuss it here.

  • Application layer: Handles conflicts

When users interact with the whiteboard, conflicts inevitably occur. But we have record and playback requirements, so we can’t allow forking of the state after a conflict. So how do we solve this thorny problem?

As shown in the figure, we introduced the concept of a server adjudicator. I let the conflict happen naturally first, allowing forking of the render content in a short time. Incorrect renderings will be corrected when the adjudicator determines the outcome of the conflict.

This model solves two key problems simultaneously:

Real-time: no need to wait for the ruling results to be synchronized, real-time.

Strong consistency: Content forks only briefly at the millisecond level, has little impact on the experience, and is easy to persist.

We simply sum up the above ideas as: deal with each first, and then correct the ruling. This approach is also widely used in the gaming industry, such as Blizzard’s Overwatch.

  • Web-based cross-platform synchronization

The Web has always been both loved and hated. Developers covet the convenience and uniformity of the Web, but fear the complexity of the Web ecosystem with its diverse browsers. After repeated comparison, we still choose to do it based on Web. Let’s expand and analyze the reasons below.

  • The core technology foundation is sound

The communication bottom layer, framework ecology, performance support and other aspects of the Web are developing more and more perfect, the current foundation is enough to support the production of various real-time interactive applications. Now well-known collaboration apps like Figma and Miro are designed for the Web, and our interactive whiteboard is no exception. Here is a schematic of our Web-SDK design.

Communication: Web Socket was popularized after 2013 and has the condition of long connection communication.

Rendering: WebGL was basically popularized after 2014, and it has the conditions to use GPU to render complex graphics.

Computing: WASM has been supported by major browsers since 2017, enabling efficient computing (Figma is widely used in the project).

Ecology: The popularity of front-end engineering frameworks such as Vue and React, and state-driven rendering models facilitate the writing of interactive applications.

  • Cross-platform maintenance costs are low

If we use native writing in each end, only common client we need to write iOS, Android, Web, Windows, Mac five sets of code, which also need to consider the problem of various platform rendering. This situation often leaves developers with little time to deal with product experience issues and a lot of time to spend on compatibility and upgrades.

But with the Web, you don’t need to worry about the above issues; each platform has its own relatively complete WebView. This allows collaboration apps to focus on improving their product experience (even so, many collaboration apps don’t support mobile devices).

  • Upgrade convenient

The design idea of interactive whiteboard is more similar to games. Those who play online games often know that different versions of games cannot be served together. Therefore, the upgrade of interactive whiteboard products is often forced to upgrade the whole series, so the Web is easier to upgrade, after all, there is no need to download a software installation package so troublesome.

How will interactive whiteboards evolve in the future

Above, we have analyzed the use scenario and technical composition of the interactive whiteboard, and roughly understood the role and development status of such products at present. So how will the whiteboard evolve?

I believe that with the further development of the industry, the interactive whiteboard will be divided into two main directions: one is the whiteboard used in the teaching system, the other is the whiteboard used in the conference system. The difference between these two directions may seem to be a business difference, but it can actually lead to a considerable difference in technology development path. Let’s look at each of these directions.

  • Direction of Online teaching

Agora Flat Open Source Classrooms:flat-web.whiteboard.agora.io

Whiteboards in teaching and training scenarios are mainly for teachers to output knowledge, and students’ feedback and interaction should be taken into account. The display form of this kind of whiteboard system is mainly windowed, and the interaction is similar to the desktop operating system. Use mature interactive habits, convenient display of rich information content.

We are familiar with the classroom is usually composed of two parts: class and after-class. Part of the class needs blackboard writing, courseware materials, interactive questions and answers, discipline management and auxiliary tools. The after-class section should be played back and the lecture notes should be exported.

There is also a one-to-one correspondence between the two parts of online classes:

  • Writing on the board: the whiteboard writing experience is required, with smooth and sharp strokes.

  • Data presentation: IT is necessary to convert PDF and PPT into formats that can be displayed in the whiteboard, and it is necessary to retain animation and restore typesetting as far as possible, and support the mixed use of multiple courseware.

  • Interactive question and answer and discipline management: the timer regulates the time for answering, and the buzzer improves the fun of class; Raise your hand to speak, silence management.

  • Auxiliary tools: science and technology disciplines such as mathematics and physics need to provide subject tools, such as drawing triangles, drawing coordinate systems and functions.

  • Class notes: support recording playback and blackboard export.

  • Online Meeting Directions

Miro:miro.com

In a meeting scenario, it is convenient for participants to express and record their opinions and output content through convenient tools. The main manifestation of this whiteboard is the ability to display a variety of materials on an infinite canvas, using a variety of tool plug-ins for collaborative production of content.

Offline meetings are mainly composed of the following three parts:

  • Presentation stage: The sponsors show PPT to explain their ideas and plans first.

  • Discussion stage: Participants and sponsors discuss and improve the plan (usually this stage requires whiteboard assisted communication).

  • Summary stage: Record the results of the discussion and arrange follow-up work.

When meetings move online, our requirements correspond to technology:

  • Presentation: PPT should be converted into a format that can be displayed on the whiteboard, and animation and typesetting should be preserved as much as possible.

  • Discussion aid: provides remarks, brain map, flow chart, lane map and small stickers.

  • Content output: generally summarize the content into emails according to whiteboard records, and export pictures as original data at most.

Through the comparison above, we can see the similarities and differences between many teaching and meeting scenarios. In the future, both of them will develop a variety of scene tools based on the whiteboard system to assist teaching and meeting. The difference is that teaching will continue to invest more energy in product details such as content recording and reviewing and classroom permission management.

Summary and reflection

What is the essence of an interactive whiteboard? Is that we need a better auxiliary expression tool when we work and study online. Interactive whiteboards tend to start out as a board for interactive drawing and writing, and over time they become more and more like an online operating system. Because each niche may need its own online auxiliary presentation tool, the interactive plug-in ecosystem will flourish. Understanding this helps us understand why both Figma and Miro are worth $10 billion.

If we believe that remote working and learning will become more common in the future, then the interactive whiteboard that is integral to this scenario must have ample opportunity to grow.

About the author: Wu Shuang, head of the interactive application innovation department of Sonnet, mainly responsible for the interactive whiteboard related business of sonnet. He founded Netless interactive whiteboard company, which was merged into Sonnet at the end of 2020. Keen to provide useful technical products and open source projects to developers.