Unity and Flutter can both draw their interfaces using OpengL and Vulkan. Is there a way to merge the two interfaces, rendering the flutter interface into Unity objects, or rendering the Unity interface into the Flutter widget? Since the two rendering methods are generally the same, we will focus on how to render the Flutter interface into Unity.

The first thing we thought of was taking screenshots of the Flutter interface into a bitmap, then passing the bitmap to Unity through interaction and using the bitmap in Unity, and we could immediately see that Flutter had screenshots. However, we will immediately find the disadvantages of this solution. This solution requires the Flutter interface to be downloaded from the GPU into memory, then the Bitmap is transferred to Unity through unity’s communication with Java, and finally the Bitmap is uploaded to the GPU in Unity for use as a texture. This requires a lot of shuffling around in memory, and bitmaps are often large enough to be inefficient. Is there a better way to solve this problem? Of course there is

Texture Shared

We know that OpengL context is usually bound to threads, different contexts are relatively independent and cannot share content, but in order to better work in a multi-threaded environment, OpengL provides a way to share texture to compensate for the problem, improve the efficiency of multi-threaded work. To create an OpengL environment, pass in an existing OpengL context and configs so that both OpengL environments can share the same texture (search for OpengL texture sharing).

As for how to implement Shared with the unity of the texture on the android, you can refer to the article blog.csdn.net/jnlws/artic…

This article explains in detail and the code is quite complete. You can read it carefully and implement it yourself. Here I will briefly explain the implementation process of camera texture sharing (because the flutter sharing process is similar).

  1. Call back to Java by communicating with Android in unity thread
  2. Get the Unity thread opengL context and parameter configuration in the Java method, and create a New Java thread used as the Java rendering thread, and create a New OpengL context in the Java thread, passed in unity opengL context, in order to achieve texture sharing
  3. Output android camera data to the Surfacetexture, bind that Surfacetexture to a newly created Textureid in OpengL, Render camera data to a new Textureid using fBO off-screen rendering (GL_TEXTURE_EXTERNAL_OES) So it needs to be converted to a texture format that Unity can use via off-screen rendering), returns the new Textureid to Unity
  4. Unity receives the Textureid and renders it to this GameObject

The key code is as follows

Texture sharing is an OpengL method. It is not necessary for vulkan and Metal rendering pipelines that are naturally multithreaded, but I am not familiar with Vulkan so I will not explain it here.

The Flutter interface is rendered into the texture

We have already analyzed how to share camera data to Unity via textures. We know that textures can be shared to Unity via Surfacetexture and FBO off-screen rendering, so we just need to figure out how to render the Flutter interface to the Surfacetexture. This allows flutter to merge with the Unity interface. Next we will analyze the Flutter source code, which is based on Flutter1.16.

First of all, in the new version of Flutter, in order to facilitate mixed access, the Flutter engine is removed, and we do not need to render the Flutter interface directly. So we just need to create a Flutter Fragment without a View and put it into Unity Activity. We will flutter fragments source copies and transformation, and the matching will need FlutterActivityAndFragmentDelegate, we follow the code analysis how the flutter interface rendering on the android, Flutter is first loaded into the Android interface via FlutterView

FlutterView is divided into two modes. We only need to analyze the surface mode, and we find that there is a FlutterSurfaceView to realize the association with flutter. We found that the SurfaceView surface was created by passing the surface into flutter via FlutterRender.

This analysis makes it clear that we just need to create a surface, pass it to flutter via FlutterRender, export it to the Surfacetexture, and export it to a texture ID available to Unity via fBO off-screen rendering. The following code

  public void attachToFlutterEngine(FlutterEngine flutterEngine) {
        this.flutterEngine = flutterEngine;
        FlutterRenderer flutterRenderer = flutterEngine.getRenderer();

        flutterRenderer.startRenderingToSurface(GLTexture.instance.surface);
        flutterRenderer.surfaceChanged(GLTexture.instance.getStreamTextureWidth(), GLTexture.instance.getStreamTextureHeight());
        FlutterRenderer.ViewportMetrics viewportMetrics = new FlutterRenderer.ViewportMetrics();
        viewportMetrics.width = GLTexture.instance.getStreamTextureWidth();
        viewportMetrics.height = GLTexture.instance.getStreamTextureHeight();
        viewportMetrics.devicePixelRatio = GLTexture.instance.context.getResources().getDisplayMetrics().density;
        flutterRenderer.setViewportMetrics(viewportMetrics);
        flutterRenderer.addIsDisplayingFlutterUiListener(new FlutterUiDisplayListener() {
            @Override
            public void onFlutterUiDisplayed(a) {
                GLTexture.instance.setNeedUpdate(true);
                GLTexture.instance.updateTexture();
            }

            @Override
            public void onFlutterUiNoLongerDisplayed(a) {}}); GLTexture.instance.attachFlutterSurface(this);
    }
Copy the code

Note that the width and height need to be passed to the flutter, otherwise the flutter interface may not appear

Next we just need to receive the texture in Unity and render it to GameObject just like the camera.

The final result is as follows. We can see that the Flutter interface has been rendered perfectly into Unity

The overall code flow is as follows. During the redrawing, the data are all inside the GPU, and no back and forth copy is involved. This is the benefit of texture sharing, which can achieve a higher refresh rate.

Click on the event

We then need to process the flutter click event. We need to get the flutter click event in Unity and pass it to Android, and then to Flutter.

Unity click handling code is shown below

void Update()
{
    #if UNITY_ANDROID
    if(mGLTexCtrl.Call<bool> ("isNeedUpdate"))
        mGLTexCtrl.Call("updateTexture");
    if (Input.touches.Length > 0) {if(haveStartFlutter == 1) {// Coordinates are converted and passed to Java, code omitted, see Unity event handling for details
        }else{
            mFlutterApp.Call("startFlutter");
            haveStartFlutter = 1; }}#endif 

    if(Input.GetMouseButtonDown(1)){ Debug.Log(Input.mousePosition); }}Copy the code

Android passes the code to Flutter as follows. Here we can find how to pass click events in the Flutter source, which is also done with the flutter source

public void onTouchEvent(int type, double x, double y) {
        ByteBuffer packet =
                ByteBuffer.allocateDirect(1 * POINTER_DATA_FIELD_COUNT * BYTES_PER_FIELD);
        packet.order(ByteOrder.LITTLE_ENDIAN);
        double x1, y1;
        x1 = GLTexture.instance.getStreamTextureWidth() * x;
        y1 = GLTexture.instance.getStreamTextureHeight() * y;
        Log.i("myyf"."x:" + x1 + "&y:" + y1 + "&type:" + type);
        addPointerForIndex(x1, y1, type + 4.0, packet);
        if(packet.position() % (POINTER_DATA_FIELD_COUNT * BYTES_PER_FIELD) ! =0) {
            throw new AssertionError("Packet position is not on field boundary");
        }

        flutterEngine.getRenderer().dispatchPointerDataPacket(packet,packet.position());

    }

// To pass click information, refer to flutter
// TODO(mattcarroll): consider creating a PointerPacket class instead of using a procedure that
// mutates inputs.
private void addPointerForIndex(
        double x, double y, int pointerChange, int pointerData, ByteBuffer packet) {
    if (pointerChange == -1) {
        return;
    }
    / /... Is omitted
   

}
Copy the code

This allows us to click on the Flutter screen in Unity and achieve the following effect

conclusion

We have analyzed how to render the Interface of flutter into Unity through OpengL texture sharing and Android Surface. Similarly, how to render the Unity interface into Flutter is the same. We just need to customize UnityPlayer to export it to the texture. We can use this method to render android interfaces into Flutter and Unity.