In the middle of this year, Google launched ARCore with much anticipation, a seamless blend of the physical and the digital to enrich our real world. Through it, developers can develop AR applications on Android platform more quickly and conveniently. With AR technology, a large number of products can find novel application scenarios, and even open up a new product line.

At present, there are many products based on AR technology in the market. For example, IKEA’s IKEA Place app provides a new online way to purchase furniture. Users only need to Place the camera of their mobile phone at the corner of the furniture they want to Place, then select the furniture they want, and complete the layout by simple dragging and rotating. Check to see if this piece of furniture suits your wishes.

Below is a diagram of how to use IKEA Place, and it looks like the chair will fit 🙂

So if AR is combined with other technologies, will there be more exciting application scenarios?

Qiniu Real-time audio and video Cloud (hereinafter referred to as Qiniu RTN) is based on the widely standardized WebRTC technology stack, with full platform compatibility, supporting major browsers such as Chrome, Safari, Firefox and Android, iOS, Windows, etc. The powerful Qiniu real-time audio and video streaming media network has more than 180 data centers around the world, with powerful link acceleration function and abundant nodes to ensure that customers can get acceleration no matter where they are distributed in the world. The average ultra-low delay of 200ms provides the most fundamental support for many customer scenarios with demanding real-time requirements, such as one-to-one chat, chat room, video conference, online education and other scenarios with strong demand for interaction are very suitable for using quniu RTN.

In this article, we will integrate AR technology into real-time audio and video calls with Google’s official example hello_AR_java, which will be applied to the 1.1.0+ version of quniu RTN SDK’s new function “external audio and video data import”.

The following is a GIF of the effect

Preparation 0: Integrate qiniu RTN SDK into AR Demo

Before we can actually start coding, we need to get the project and environment built

downloadSeven cattle RTN SDKGo to current directoryQNRTC-Android

git clone [email protected]:pili-engineering/QNRTC-Android.git
Copy the code

Download ARCore to the current directoryarcore-android-sdk

git clone [email protected]:google-ar/arcore-android-sdk.git
Copy the code

Copy the corresponding qiniu RTN SDK file to hello_ar_java project

  1. The fileQNRTC - Android/releases/qndroid - RTC - 1.2.0. JarCopy toarcore-android-sdk/samples/hello_ar_java/app/libs/(The liBS directory needs to be created by yourself)
  2. willQNRTC-Android/releases/Under theArmeabi, ArmeabI-V7A, ARM64-V8A, x86Wait for the four folders to be copiedarcore-android-sdk/samples/hello_ar_java/app/src/main/jniLibsIn folder (jniLibs directory needs to be created by yourself)
  3. Open it using AndroidStudioarcore-android-sdk/samples/hello_ar_javaProject to modify several of the configurations
    • In order for the project to reference the libraries added in the previous two steps, openapp/build.gradleFile,dependenciesAdd lineimplementation fileTree(include: ['*.jar'], dir: 'libs')
    • In order to make a real-time call, you need to set the program to use the network permission, openAndroidManifest.xmlFile,manifestAdd the following permission statement to the tag
      • <uses-permission android:name="android.permission.INTERNET"/>
      • <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>

Introduction to core Classes

Before the actual coding and code analysis, let’s give a brief overview of the core classes that will be involved

QNRTCManager: QNIu RTN SDK core class, providing low latency real-time audio and video call capabilities

Session: ARCore core class, which manages AR system status, including Camera collection, point-network monitoring, and plane detection

GLSurfaceView & Renderer: The View class and render class provided by the Android system are responsible for screen display and rendering respectively

BackgroundRenderer & ObjectRenderer & PlaneRenderer & PointCloudRenderer: Render classes provided in the Demo that are responsible for rendering the following sections, respectively

  • Background rendering (camera preview raw image)
  • Object and Shadow Rendering (Android model and Shadow)
  • Plane rendering (plane detected by AR system)
  • Point cloud rendering (point cloud detected by AR system)

Preparation 1: Establish a basic real-time audio and video call environment

First of all, we need to implement the real-time audio and video room event listener, QNRoomEventListener, which needs to be implemented in many ways, the following is only to show this simple example to use the method, the complete interface description is here

public class HelloArActivity extends AppCompatActivity implements GLSurfaceView.Renderer.QNRoomEventListener {
    private boolean mPublished = false; // Indicates whether the local publication is successful.@Override
    public void onJoinedRoom(a) {
    	mRTCManager.publish(); // After joining the room successfully, try to publish
    }

    @Override
    public void onLocalPublished(a) {
    	mPublished = true; // After successful publishing, identify it as true}... }Copy the code

At the end of the onCreate method, initialize the real-time audio and video call environment and add the specified Room. Here is how to obtain the Room Token

protected void onCreate(Bundle savedInstanceState) { ... QNRTCSetting setting = new QNRTCSetting(); setting.setExternalVideoInputEnabled(true); / / open external video mRTCManager import. SetRoomEventListener (this); Mrtcmanager.initialize (this, setting); // RTN SDK initializes mRTCManager. JoinRoom (###Your Room Token###); // Use Room Token to join a Room}Copy the code

Preparation 2: Establish the basic AR environment

Use GLSurfaceView & Renderer to prepare for drawing AR images

Implement the GlsurfaceView.renderer interface in the Activity class declaration, as shown in this Demo, and then we need to implement three corresponding methods, the meanings of which are described in the comments

public class HelloArActivity extends AppCompatActivity implements GLSurfaceView.Renderer.QNRoomEventListener {
    /** * displays the Surface creation completion callback **/
    public void onSurfaceCreated(GL10 gl, EGLConfig config) {}.../** * callback **/ when the Surface size changes
    public void onSurfaceChanged(GL10 gl, int width, int height) {}.../** * displays the Surface creation completion callback **/
    public void onDrawFrame(GL10 gl) {}}Copy the code

Once we have implemented the Renderer rendering class, we need to provide a Surface for display so that the Renderer can render on it. GLSurfaceView has this capability.

The following sample code parses the GLSurfaceView from the layout XML file and sets up the Renderer

surfaceView = findViewById(R.id.surfaceview); // Parse the GLSurfaceView from the layout XML. surfaceView.setRenderer(this); / / set the Renderer
Copy the code

Create a Session

Session is the main entry class of the AR system and must be initialized and started before any AR operation

protected void onResume(a) {
    session = new Session(/* context= */ this); // AR system initialization. session.resume();/ / AR session, try to open the camera, such as cameras occupied, throws CameraNotAvailableException anomalies
}
Copy the code

Use OpenGL Shader to draw AR enhancement images on the display Surface

After the AR session starts, each frame of the camera provides the following information

  • Raw camera preview data
  • Flat array detected
  • Array of detected point clouds
  • Flat touch event

We can use the above events in the onDrawFrame method for corresponding processing. For example, when we encounter a plane touch event, we can place an Android model in the corresponding position and draw the detected plane and point cloud at the same time.

// Draw the background
private final BackgroundRenderer backgroundRenderer = new BackgroundRenderer();
// Draw the object
private final ObjectRenderer virtualObject = new ObjectRenderer();
// Draw an object shadow
private final ObjectRenderer virtualObjectShadow = new ObjectRenderer();
// Draw plane
private final PlaneRenderer planeRenderer = new PlaneRenderer();
// Draw cloud points
private final PointCloudRenderer pointCloudRenderer = new PointCloudRenderer();

public void onDrawFrame(GL10 gl) {
    frame = session.update(); // Get the camera's original data frame (block method)
    
    // Handle one tap per frame.
    handleTap(frame, camera); // Check if there are flat click events, if so, place Android model in the corresponding position.// Draw background.
    backgroundRenderer.draw(frame); // Draw the camera preview data as the background image.// Visualize tracked points.
    PointCloud pointCloud = frame.acquirePointCloud();
    pointCloudRenderer.update(pointCloud);
    pointCloudRenderer.draw(viewmtx, projmtx); // Draw the point cloud.// Visualize planes.
    planeRenderer.drawPlanes(session.getAllTrackables(Plane.class), camera.getDisplayOrientedPose(), projmtx); // Draw plane.// Update and draw the model and its shadow.
    virtualObject.updateModelMatrix(anchorMatrix, scaleFactor);
    virtualObjectShadow.updateModelMatrix(anchorMatrix, scaleFactor);
    virtualObject.draw(viewmtx, projmtx, colorCorrectionRgba, coloredAnchor.color); // Draw the Android model
    virtualObjectShadow.draw(viewmtx, projmtx, colorCorrectionRgba, coloredAnchor.color); // Draw the shadow of the Android model
}
Copy the code

## Technology combination: Release the AR enhanced picture to the real-time audio and video cloud

Having implemented basic live audio and video calls and AUGMENTED AR graphics, it’s now just a matter of putting them together.

Since Session will occupy the camera of the device after starting, qiniu RTN SDK cannot collect data. At this time, we need to use the function “External audio and video data import” provided by the latest version.

Before releasing the stream, we need to obtain the NV21 format data of AR enhanced images, because the current qiniu RTN Android SDK “external video data import” function only supports NV21 format data.

The following sample code is added at the end of the onDrawFrame method, reads the Surface content data from the GLSurfaceView, performs the necessary formatting transformations, and then publishes it

public void onDrawFrame(GL10 gl) {...if (mPublished) { // Import AR data only after qiniu RTN publishes stream successfully
        // Read the augmented image data from the GPU
        GLES20.glReadPixels(0.0, mSurfaceWidth, mSurfaceHeight, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, mBufferRGBA);

        // RGBA to NV21 (not expanded here for space reasons)
        mBufferNV21 = RGBAToNV21(mBufferRGBA, mSurfaceWidth, mSurfaceHeight);

        // Use the "external video Data import" function to publish the ENHANCED IMAGES of NV21 data
        mRTCManager.inputVideoFrame(mBufferNV21, mSurfaceWidth, mSurfaceHeight, 0, frame.getTimestamp()); }}Copy the code

conclusion

AR can be easily combined with real-time audio and video communication by using the “external audio and video Data import” function provided by qiniu RTN SDK 1.1.0+ version. The above program is based on qiniu RTN SDK and the corresponding RTN network, and can support up to 20 people to make low delay audio and video calls at the same time. It is believed that the combination of AR technology and real-time audio and video communication will bring more application scenarios in the near future.

Free time limit gift activities

Qiniu real-time audio and video cloud from October 30, the implementation of monthly free time limit gift activities. Free 5000 minutes for pure audio, STANDARD DEFINITION, HIGH definition and ultra clear 4 gears respectively. If fully used, the total consumption amount will be 770 YUAN.