This article was first published at:Walker AI
Recently, I have been working on the cloud platform for Android devices. I believe that the device cloud platform is not unfamiliar to you. Common functions such as remote monitoring and operation of a mobile phone, uploading files, and checking logs are used. But when you want to use the cloud phone to watch video, play games, listen to music, you will find that the current market of the cloud phone, no one has a voice transmission function.
After discovering this situation, I came up with the idea of making this feature. What should I do specifically? Before I did this, I did a little research to see if there were any existing solutions that I could refer to.
After several days of research, let me tell you the general results first.
Voice transmission is limited, but still achievable.
1. Comparison of existing programs
| Equipment microphone | Third party projection software | Sndcpy | |
|---|---|---|---|
| The sound source | Mobile phone microphone | Mobile phone microphone | The app streams its own data |
| The effect | It is subject to external interference | Will interfere, but there are also algorithms for noise reduction | Depends on the audio data itself |
| Application restrictions | Other applications call the microphone stream | Other applications call the microphone stream | An application with no permission granted |
| The android version | any | any | Android 10 + |
2. Scheme selection
In fact, it is not hard to see that the third-party screen projection software uses the microphone of the device, just because the data is first passed through the software client, before the audio is played, I use the algorithm to de-noise (burning goose algorithm de-noise is spicy chicken). But have you ever thought about its practical effect? Imagine the usage scenario as a cloud platform for devices in general:
Such intensive equipment, such a scene, by the external capture sound, a good say, more than a few, the effect who can bear it? And if the phone itself has applications that require the use of a microphone, the effect is self-evident. So, using the device microphone, as a platform, is definitely not a good solution!
Since the microphone doesn’t work, does that mean there’s no other way? My gut tells me of course not. If there was a way to capture the app’s media stream directly, instead of recording it with outgoing sound, that would be a huge boost! After discovering this, I followed the lead and found SNDcpy, an open source project.
3. Plan implementation
3.1 Project Introduction
Sndcpy making address
There is a detailed description in the introduction of the project, here I will only make a brief overview:
(1) Only support Android 10 and above devices
(2) Cross-system platform operation
(3) You can use VLC to listen to connected devices
For the first point, which is essentially for the SDK version with the interface, the Android app’s configuration allows a switch to be turned on that allows the audio stream to be monitored, so that the app can get the sound.
(For example, QQ Music, however, after trial, Google browser is not the right kind)
3.2 Steps
(1) Download the ZIP file that has been built in the project (if there is no ADB, download the with-ADB version and configure the environment, here is not how to configure the environment);
(2) Connect the mobile device to./sndcpy in the unzip directory, with the device number after the case of multiple devices;
(3) Confirm permission on mobile phone;
(4) Download and install VLC;
(5) When running VLC, you may encounter an error for the first time, but it doesn’t matter, just try again.
4. Integrate
It can be used, but as a cloud platform project, customers need to play and listen to the device through the web page, and can only hear it through VLC, which is obviously not in line with the use requirements. As a streaming media real-time network communication, it is natural to think of making it a websocket, which requires the implementation of the existing socket service.
However, if you look closely at the documentation of this project, you will see that what we want is actually available. There are three files in the compressed package, including the.bat file. Let’s open it and have a look:
#! /bin/bash set -e ADB=${ADB:-adb} VLC=${VLC:-vlc} SNDCPY_APK=${SNDCPY_APK:-sndcpy.apk} SNDCPY_PORT=${SNDCPY_PORT:-28200} serial= if [[ $# -ge 1 ]] then serial="-s $1" echo "Waiting for device $1..." else echo 'Waiting for device... ' fi "$ADB" $serial wait-for-device "$ADB" $serial install -t -r -g "$SNDCPY_APK" || { echo 'Uninstalling existing version first... ' "$ADB" $serial uninstall com.rom1v.sndcpy "$ADB" $serial install -t -g "$SNDCPY_APK" } "$ADB" $serial forward tcp:$SNDCPY_PORT localabstract:sndcpy "$ADB" $serial shell am start com.rom1v.sndcpy/.MainActivity echo "Press Enter once audio capture is authorized on the device to start playing..." read dummy "$VLC" -Idummy --demux rawaud --network-caching=0 --play-and-exit tcp://localhost:"$SNDCPY_PORT"It was discovered that the TCP service was started using the ADB command for the transmission. In order to facilitate you to operate directly, here I will directly list the commands and steps. If you follow these steps, the problem of Android-terminal device transmission can be solved:
(1) put sndcpy.apk in /data/local/ TMP directory, it is recommended to use adb command adb push sndcpy.apk /data/local/ TMP
(2) adb forward TCP :port localabstract:sndcpy
(3) the adb shell am start com. Rom1v. Sndcpy /. MainActivity
(4) Locally start a socket, bind the first port defined, if the data returned is not empty, congratulations, you succeeded! The basic startup is like this, but for a Web project, the socket is forwarded once to WebSocket to complete the playback on the Web side (of course, also need the front end itself to support such a format of audio streaming processing).
And that’s it!