This is the 12th day of my participation in the August More Text Challenge. For details, see:August is more challenging

Android Screen sharing – transfer pictures

This article mainly explains how to transfer screenshots between Android and Android, which belongs to the introduction of Android screen sharing. Application scenario: Android screen sharing tool.

First, the effect diagram

rendering

The demo interface

Second, the preface

At present, there are many kinds of software to realize android screen sharing. The root of android screen sharing is nothing more than the process of collection, transmission and playback. No matter whether it is screenshot transmission or hard decoding, Android must be inseparable from core MediaProjection, which is the video recording interface (screen acquisition interface) opened by Google.

Examples of good software:

  • TeamViewer

  • sunflower

  • Vysor

Vysor is through ADB to achieve no root silent screenshots, and then transmission, computer control mobile phone. In the whole process of screen sharing, the focus is on how to collect, collect clear picture quality and small memory technology becomes the core.

Common Android terminal acquisition techniques include:

  • MediaProjection implementation screenshot

  • MediaProjection hard decoding

  • Ffmpeg soft decoding

  • Adb implementation no root screenshot

Three, function points

  1. Android5.0 and above do not require root screenshots

  2. Collect when the Android screen changes

  3. Multi-end sharing extension is simple

  4. The socket transmits an array of bytes

  5. Use native code implementations whenever possible

Four, function explanation

4.1 screenshot

With the continuous upgrade of Android system, the permissions are increasingly tightened. Silent screenshots cannot be done without root and data cable connection, which refers to the interception of areas that do not belong to the app.

  • Under the condition that the data line is allowed to be connected, adb shell can push the pre-compiled dex file to the phone to achieve silent screenshots.

  • In the case of no linked data line and no root, the interception can only be explicitly made aware by the user. In this screenshot, MediaProjection is used, as shown below:

MediaProjection is an Android 5.0 open interface for screen capture and video recording. It is a system-wide service that dynamically requests permissions before use and is processed in onActivityResult.

eg:

Private void tryStartScreenShot() {MediaProjectionManager mProjectionManager = (MediaProjectionManager)  getSystemService(Context.MEDIA_PROJECTION_SERVICE); if (mProjectionManager ! = null) { startActivityForResult(mProjectionManager.createScreenCaptureIntent(), REQUEST_MEDIA_PROJECTION); }}Copy the code
@Override protected void onActivityResult(int requestCode, int resultCode, Intent data) { if (requestCode == REQUEST_MEDIA_PROJECTION && data ! = null) {if (resultCode == RESULT_OK) {// ScreenShotHelper ScreenShotHelper = new ScreenShotHelper(this, resultCode, data, this); screenShotHelper.startScreenShot(); } else if (resultCode == RESULT_CANCELED) {logWrapper. d(TAG, "user canceled "); }}}Copy the code

While screensharing, taking screenshots all the time is performance intensive, it would be nice to take screenshots when the screen changes.

Just right, in the ImageReader setOnImageAvailableListener can make us convenient to change the screen data. Eg:

/ * * * change screen capture * / private class ImageAvailableListener implements ImageReader. OnImageAvailableListener {@ Override public void onImageAvailable(ImageReader reader) { try (Image image = reader.acquireLatestImage()) { if (image ! = null) { // todo } } catch (Exception e) { e.printStackTrace(); }}}Copy the code

4.2 Image Processing

Image format:

  • PNG: lossless compressed image format, support Alpha channel

  • Jpeg: Lossy compressed image format without background transparency

  • Webp: “WebP is a relatively new image format supported by Android 4.2.1 (API level 17). This format provides excellent lossless and lossy compression for images on the web. With WebP, developers can create smaller, richer images. WebP lossless image files are on average 26% smaller than PNG files. These image files also support transparency (also known as Alpha channels) by adding only 22% bytes.

WebP lossy images are 25-34% smaller than the same JPG images using the equivalent SSIM quality index. For cases where lossy RGB compression is acceptable, lossy WebP also supports transparency, and the resulting file size is typically three times smaller than PNG.” — from the official Google docs

I have tried webP format encoding, the size is reduced by 50% or so, but the coding time is 100% longer, equivalent to CPU for network speed. It seems that WebP is not suitable for this scenario.

Image compression:

  • Compressed size

  • Compressed image quality

The Demo code is to compress the bitmap to a fixed size

Server:

/** * Compress the image (compressed does not represent the actual size, * * @param bitmap compressed image * @param sizeLimit sizeLimit unit k * @return compressed image */ public static bitmap compressBitmap(Bitmap bitmap, long sizeLimit) { ByteArrayOutputStream baos = new ByteArrayOutputStream(); int quality = 90; bitmap.compress(Bitmap.CompressFormat.JPEG, quality, baos); While (baos.tobyteArray ().length / 1024 > sizeLimit) {baos.reset(); bitmap.compress(Bitmap.CompressFormat.JPEG, quality, baos); quality -= 10; } return BitmapFactory.decodeStream(new ByteArrayInputStream(baos.toByteArray()), null, null); }Copy the code

4.3 the socket transmission

Native sockets send large amounts of data, will be subcontracted, when receiving the packet must be combined;

While there are some third-party socket frameworks that can help us handle the operation of combining packages, WebSocket handles this issue in its protocol standard.

So the Demo uses Websocket for communication. Just put in the data from the screenshots you got.

import com.talon.screen.quick.util.LogWrapper; import org.java_websocket.WebSocket; import org.java_websocket.handshake.ClientHandshake; import org.java_websocket.server.WebSocketServer; import java.net.InetSocketAddress; /** * @author by Talon, Date on 2020-04-13. * note: Websocket server */ public class extends WebSocketServer {private final String TAG = "MWebSocketServer"; private WebSocket mWebSocket; private boolean mIsStarted = false; private CallBack mCallBack; public MWebSocketServer(int port, CallBack callBack) { super(new InetSocketAddress(port)); this.mCallBack = callBack; setReuseAddr(true); setConnectionLostTimeout(5 * 1000); } @override public void onOpen(WebSocket WebSocket, ClientHandshake handshake) {logWrapper.d (TAG, "with user link "); mWebSocket = webSocket; } @override public void onClose(WebSocket conn, int code, String Reason, Boolean remote) {LogWrapper. D (TAG, "user left "); } @override public void onMessage(WebSocket conn, String message) {logWrapper. e(TAG, "received message:" + message); } @override public void onError(WebSocket conn, Exception ex) {LogWrapper. E (TAG, "Error :" + ex.toString()); } @Override public void onStart() { updateServerStatus(true); } /** * Stop the server */ public void socketStop() {try {super.stop(100); updateServerStatus(false); } catch (InterruptedException e) { e.printStackTrace(); }} /** * Send binary ** @param bytes */ public void sendBytes(byte[] bytes) {if (mWebSocket! = null) mWebSocket.send(bytes); } private void updateServerStatus(boolean isStarted) { mIsStarted = isStarted; LogWrapper.e(TAG, "mIsStarted:" + mIsStarted); If (mCallBack! = null) mCallBack.onServerStatus(isStarted); } public boolean isStarted() { LogWrapper.e(TAG, "mIsStarted:" + mIsStarted); return mIsStarted; } public interface CallBack { void onServerStatus(boolean isStarted); }}Copy the code

Client:

import com.talon.screen.quick.util.BitmapUtils; import com.talon.screen.quick.util.LogWrapper; import org.java_websocket.client.WebSocketClient; import org.java_websocket.handshake.ServerHandshake; import java.net.SocketException; import java.net.URI; import java.nio.ByteBuffer; /** * @author by Talon, Date on 2020-04-13. * note: Websocket client */ public class extends WebSocketClient {private final String TAG = "MWebSocketClient"; private boolean mIsConnected = false; private CallBack mCallBack; public MWebSocketClient(URI serverUri, CallBack callBack) { super(serverUri); this.mCallBack = callBack; } @Override public void onOpen(ServerHandshake handshakeData) { LogWrapper.e(TAG, "onOpen"); updateClientStatus(true); try { getSocket().setReceiveBufferSize(5 * 1024 * 1024); } catch (SocketException e) { e.printStackTrace(); } } @Override public void onMessage(String message) { } @Override public void onMessage(ByteBuffer bytes) { byte[] buf =  new byte[bytes.remaining()]; bytes.get(buf); if (mCallBack ! = null) mCallBack.onBitmapReceived(BitmapUtils.decodeImg(buf)); } @Override public void onClose(int code, String reason, boolean remote) { updateClientStatus(false); } @Override public void onError(Exception ex) { updateClientStatus(false); } private void updateClientStatus(boolean isConnected) { mIsConnected = isConnected; LogWrapper.d(TAG, "mIsConnected:" + mIsConnected); If (mCallBack! = null) mCallBack.onClientStatus(isConnected); } public boolean isConnected() { LogWrapper.d(TAG, "mIsConnected:" + mIsConnected); return mIsConnected; } public interface CallBack { void onClientStatus(boolean isConnected); void onBitmapReceived(Bitmap bitmap); }}Copy the code

4.4 Demo Download Address

Download the Demo