Audio net SDK access and audio and video call application development guide

1. Introduction

In today’s mobile Internet era, audio and video related applications have penetrated into every aspect of our daily life.

And the audio and video technology behind it is also very mature, short video editing, live with goods, video voice calls and other applications all reflect the convenience of audio and video technology to our life.

And this time to have a practical experience, access to the audio SDK of sonnet, and create an audio and video call application.

2. Account registration and project creation

The first thing to do is to register an account at www.agora.io/cn and complete the real-name verification.

Once you’re logged in, you’re in the admin screen, and there’s a bunch of options on the left that you can do, so go directly to project management and create our VideoChat project.

In the security mode of project creation, choose the APPID + Token mode. In order to improve the security of the project, Agora will cancel the support for APP ID authentication scheme in the future.

When the project is created, it will have the corresponding APP ID, which will be used in the subsequent code development.

In addition, we also need to generate tokens, in the project management page to generate a temporary Token.

This temporary Token is time-limited and is only for testing, and if you have your own back-end development, you can generate the formal Token in the production environment. Also, the Token string will be used in future development.

3. SDK download and integration

Once the project is created and the APP ID and Token are ready, you can download the audio SDK from the official website.

Download the decompressed content as shown above. The libs folder contains the so dynamic library corresponding to different CPU architectures, as well as the corresponding header file of the dynamic library and the Jar package of the Java version, which can be integrated with the C++ version and the Java version respectively.

Here, I only use the Java version to demonstrate. In fact, many methods in the Java version are called in Native, and native calls are methods in the SO dynamic library. With the C++ version of integration, you also need to write your own JNI code to connect Java to C++, which is not as fast as using the Java version directly.

There are two ways to integrate SDK. One is to use JCenter directly to integrate SDK. Just add a line of code to build.gradle.

dependencies {
    implementation 'the IO. Agora. RTC: full - SDK: 3.1.3'
}
Copy the code

The version number can be found on the official website. The latest version is 3.1.3.

Of course, you can also integrate agora-RtC-SDK. jar and all versions of SO into the corresponding directory by loading the JAR package and so dynamic library, as shown in the picture below:

Also don’t forget to add the following code to build. Gradle in your app directory:

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
}
Copy the code

Without this line of code, the added JAR package cannot be resolved.

Above, completed the whole SDK project access, the next is the code development link.

4. Application development

What we are going to develop is an audio and video calling application, just like wechat video, think about what the content will be.

First of all, we need to collect our pictures through the Camera, and then record our voice through the microphone, and then transmit it to the other party through the network. In addition, we can hear the voice of the other party and display the picture on the screen.

If you think about all of this, it’s a lot of work that four or five people might not be able to handle, but with professional audio and video SDKS, one person can do most of the work.

The next step is to complete the development work. The code is available on Github:

Github.com/glumes/agor…

4.1 Permission Application

The first is permission application, which involves Camera, network, storage, recording and so on. The specific permission list is also given on the official website, as follows:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
   package="io.agora.tutorials1v1acall">

   <uses-permission android:name="android.permission.READ_PHONE_STATE" />   
   <uses-permission android:name="android.permission.INTERNET" />
   <uses-permission android:name="android.permission.RECORD_AUDIO" />
   <uses-permission android:name="android.permission.CAMERA" />
   <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
   <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
   <uses-permission android:name="android.permission.BLUETOOTH" />
   <uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
   // If your scenario involves reading external storage, add the following permissions:
   <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
   // If you are running Android 10.0 or above, you need to add the following permissions:
   <uses-permission android:name="android.permission.READ_PRIVILEGED_PHONE_STATE" />
...
</manifest>
Copy the code

Github:

Github.com/glumes/agor…

4.2 Engine Creation

The next step is to create the RtcEngine, through which many of the SDK’s methods are called.

/ / declare RtcEngine
private RtcEngine mRtcEngine;
/ / create RtcEngine
private void initializeEngine(a) {
    try {
        mRtcEngine = RtcEngine.create(getBaseContext(), getString(R.string.agora_app_id), mRtcEventHandler);
    } catch (Exception e) {
        Log.e(TAG, Log.getStackTraceString(e));
        throw new RuntimeException("NEED TO check rtc sdk init fatal error\n" + Log.getStackTraceString(e)); }}Copy the code

The APP ID is required when creating the RtcEngine, as mentioned earlier, and is already available when creating the project.

The last parameter is mRtcEventHandler, which is an abstract class of type IRtcEngineEventHandler that defines methods to respond to RtcEngine’s various state call-back, such as a successful connection. Connection failure, joining channel, leaving channel, network state change, etc. If you need to do some monitoring of your application’s performance, you can do statistics and buried points in the callback method of IRtcEngineEventHandler.

4.3 Realize Camera preview

RtcEngine encapsulates Camera related operations in the SDK, through a few lines of code can achieve Camera preview.

4.3.1 Create Camera preview control View

First, we need to have a control to accept the output content of the display Camera screen. We can use SurfaceView or TextureView, but we do not need to create the control, but use the method provided by Agora SDK.

// The scenario using SurfaceView
SurfaceView mLocalView = RtcEngine.CreateRendererView(getBaseContext());
// Scenario using TextureView
TextureView mLocalView = RtcEngine.CreateTextureView(getBaseContext());
Copy the code

TextureView was only available in version 3.1.0 of the Agora SDK. It differs from SurfaceView in that TextureView can zoom, rotate, and pan images, while SurfaceView is better suited for video calls and live events.

Once you have the View displayed, add it to the control tree of the current Activity, and the back Camera screen will be output here.

4.3.2 Configure Camera output and enable preview

The next step is to get the Camera to output the image, but you don’t have to write one line of Camera code, you can write three lines of code.

// Start the video
mRtcEngine.enableVideo();
// Initialize the local view
mRtcEngine.setupLocalVideo(new VideoCanvas(mLocalView, VideoCanvas.RENDER_MODE_HIDDEN, 0));
// Enable preview
mRtcEngine.startPreview();
Copy the code

The above code will display our picture on the screen, the default is the front camera content.

Github:

Github.com/glumes/agor…

In the code development, the video module should be enabled first, which is disabled by default and can also be disabled through disableVideo. The audio module is enabled by default. You can also use enableAudio and disableAudio to enable or disable the audio module.

After that, the setupLocalVideo method is used to initialize the local view, mainly to set the video information of the local user, that is, how to display the picture in the SurfaceView. The configuration information is delivered by VideoCanvas class, which has a variety of parameter types.

VideoCanvas constructor type
VideoCanvas (View view)
VideoCanvas (View view, int renderMode, int uid)
VideoCanvas (View view, int renderMode, String channelId, int uid)
VideoCanvas (View view, int renderMode, int uid, int mirrorMode)
VideoCanvas (View view, int renderMode, String channelId, int uid, int mirrorMode)
Copy the code

RenderMode specifies how our renderMode should be displayed in the SurfaceView.

RENDER_MODE_HIDDEN
- The priority is to ensure that the window is filled. The video size is scaled equally until the entire window is filled with video. If the length and width of the video are different from the display window, the extra video will be truncated.
RENDER_MODE_FIT
- Ensure that all video content is displayed first. The video size is scaled equally until one side of the video window is aligned with the window border. If the length and width of the video are different from that of the display window, unfilled areas of the window will be blacked out
RENDER_MODE_FILL
- The video size is scaled and stretched to fill the display window

RENDER_MODE_HIDDEN mode is always used by default. There is also a property like mirrorMode to set whether the image is mirrored or not.

Once the screen is configured, call startPreview and you can see the screen.

4.3.3 More configuration operations of Camera

The Agora SDK does a lot of wrapping, such as the call logic for Camera1 and Camera2, resolution policy for Camera output, front and rear Camera selection, etc.

Before we can open the preview setCameraCapturerConfiguration way to configure the Camera information they want.

// Configure Camera information
mRtcEngine.setCameraCapturerConfiguration(new CameraCapturerConfiguration(CAPTURER_OUTPUT_PREFERENCE_AUTO,CAMERA_FRONT));
Copy the code

Camera parameter information mainly in CameraCapturerConfiguration class, its two parameters are enumerated types, among them the second parameter specifies the use of front or rear Camera.

The first parameter is some policy about Camera output resolution. Agora SDK does not provide an interface for us to specify the width and length of Camera output. Moreover, it combines three policies according to the usage scenario, and we call them.

CAPTURER_OUTPUT_PREFERENCE_AUTO
- (Default) Automatically adjusts collection parameters.
CAPTURER_OUTPUT_PREFERENCE_PERFORMANCE
- Ensure equipment performance first.
CAPTURER_OUTPUT_PREFERENCE_PREVIEW
- Ensure the quality of video preview first.

Through the above operations, the Camera picture preview display is completed.

4.4 Adding a Call Channel

The next step is to join a channel and communicate with friends on the same channel.

// Function prototype
joinChannel	(	String 	token,String channelName,String optionalInfo,int optionalUid) 

// Specific call
private void joinChannel(a) {
    / / get the Token
    String token = getString(R.string.agora_access_token);
    // Token and channel name must match
    mRtcEngine.joinChannel(token, "demoChannel1"."".0);
}
Copy the code

Join in the joinChannel method, where the token is the one we created earlier, and the channel name is the one we specified when we created the token. The two need to match. OptionalInfo is an optional string. OptionalUid is the user ID and is optional. If it is not specified (set to 0), the SDK will automatically assign one and return it in the onJoinChannelSuccess callback method.

When we successfully join a channel, the onJoinChannelSuccess method in IRtcEngineEventHandler is called back.

With the leaveChannel method, we can leave the current channel and also call back the onLeaveChannel method in IRtcEngineEventHandler.

4.5 Audio and video data encoding format parameter setting

After joining the channel, we have to communicate with our friends in the channel. To send our pictures and sounds to each other, the data must be encoded and sent over the network.

Note: The audio and video parameters must be set before joining the channel, that is, before the joinChannel method is called. Calls after that do not take effect.

4.5.1 Video Coding Parameter Settings

Here we need to specify the resolution of the video coding, frame rate, bit rate, video and other information, through setVideoEncoderConfiguration method.

mRtcEngine.setVideoEncoderConfiguration(new VideoEncoderConfiguration(
        // Video resolution
       VideoEncoderConfiguration.VD_640x360,
        / / frame rate
       VideoEncoderConfiguration.FRAME_RATE.FRAME_RATE_FPS_15,
        / / bit rate
       VideoEncoderConfiguration.STANDARD_BITRATE,
        // Video direction
       VideoEncoderConfiguration.ORIENTATION_MODE.ORIENTATION_MODE_FIXED_PORTRAIT));
Copy the code

In earlier versions of the Agora SDK this was done using setVideoProfile, and you can also specify specific width, height, and frame rate values.

Latest version with VideoEncoderConfiguration to configuration and configuration parameters are have defined the relevant constants, don’t have to write their own 720, 1280, the Magic Number.

Parameter Settings are not set in stone, because network or performance factors will fluctuate a little, but we will still take the closest value we set.

4.5.2 Audio Coding Parameter setting

The audio parameters are set using the setAudioProfile method.

mRtcEngine.setAudioProfile(
        // Set the audio sampling rate, bit rate, encoding mode and number of channels
        Constants.AudioProfile.DEFAULT.ordinal(),
        // Set the audio application scenario
        Constants.AudioScenario.DEFAULT.ordinal());
Copy the code

Audio we can set the sampling rate, bit rate, coding mode and number of channels. As with video Settings, we don’t have to specify a specific value, Agora SDK is packaged according to the business usage scenario, and can be set as needed.

The audio AudioProfile has the following configurations:

DEFAULT
- In communication scenarios, this option specifies a sampling rate of 32 kHz, voice coding, mono, and a maximum encoding rate of 18 Kbps.
- For live broadcast scenarios, this option specifies a sampling rate of 48 kHz, music encoding, mono, and a maximum encoding rate of 64 Kbps.
SPEECH_STANDARD
- Specify 32 kHz sampling rate, voice encoding, mono, encoding rate maximum 18 Kbps.
MUSIC_STANDARD_STEREO
- Specify 48 kHz sampling rate, music encoding, mono, encoding rate maximum 64 Kbps.
MUSIC_HIGH_QUALITY
- Specify 48 kHz sampling rate, music encoding, mono, encoding rate maximum 96 Kbps.
MUSIC_HIGH_QUALITY_STEREO
- Specify 48 kHz sampling rate, music encoding, dual channel, encoding rate maximum 128 Kbps.

The application scenarios of audio are as follows:

DEFAULT
- Default audio application scenarios
CHATROOM_ENTERTAINMENT
- For entertainment applications, the mic is frequently accessed
EDUCATION
- Educational applications, fluency and stability preferred
GAME_STREAMING
- High audio quality chat room app
SHOWROOM
- Show app, sound quality first and better professional peripherals support
CHATROOM_GAMING
- The game open black

In general, we use the DEFAULT configuration DEFAULT.

4.6 User Joins and Displays the Peer screen

Once you have set your own data encoding parameters and successfully joined the channel, it is time to receive other people’s pictures and messages in the channel.

Github:

Github.com/glumes/agor…

4.6.1 Determine whether a user joins a channel

When the onUserJoined method in IRtcEngineEventHandler calls back, indicating that someone has joined the current channel, the remote user view can be created and initialized.

If the video recording function is enabled, the video recording service will also call the onUserJoined method. If a robot joins the channel, it needs to distinguish between the two. It cannot create a remote video for the robot, and the video stream will not be sent.

In order to avoid misjudgment, robot join 2.9.0 version after more advice in the callback onRemoteVideoStateChanged method to establish the remote user view.

OnRemoteVideoStateChanged method as the name suggests is when the remote user status change is invoked, which defines the following condition:

REMOTE_VIDEO_STATE_STARTING
- The local user has received the first video packet from the remote device
REMOTE_VIDEO_STATE_DECODING
- The remote video stream is being decoded and playing normally
REMOTE_VIDEO_STATE_STOPPED
- Default initial status of the remote video
REMOTE_VIDEO_STATE_FROZEN
- The remote video stream is stuck
REMOTE_VIDEO_STATE_FAILED
- Description The remote video stream fails to play

When the state is REMOTE_VIDEO_STATE_STARTING or REMOTE_VIDEO_STATE_DECODING, we determine that a friend has joined the channel, and then we establish the remote user view.

@Override
public void onRemoteVideoStateChanged(final int uid, int state, int reason, int elapsed) {
      if (state == Constants.REMOTE_VIDEO_STATE_STARTING){
           runOnUiThread(new Runnable() {
               @Override
               public void run() {
                    setupRemoteVideo(uid); }}); }}Copy the code

Including onRemoteVideoStateChanged callback methods on behalf of the state, the state parameters in the uid to identify the remote user id, if there are multiple friends joined the channel, we will use the uid to differentiate between users, to avoid the leakage or repeat setup the remote user view.

4.6.2 Creating a Remote User View

Next, create the remote user view, which is basically the same as the Camera preview view.

// Create SurfaceView by Agora SDK
mRemoteView = RtcEngine.CreateRendererView(getBaseContext());
// Add SurfaceView to the layout control tree of the current Activity
mRemoteContainer.addView(mRemoteView);
// setupLocalVideo is used for Camera preview and setupRemoteVideo is used for remote users
mRtcEngine.setupRemoteVideo(new VideoCanvas(mRemoteView, VideoCanvas.RENDER_MODE_HIDDEN, uid));     
Copy the code

The Agora SDK also creates the SurfaceView, adds it to the control container, and finally completes the creation through the setupRemoteVideo method. The whole logic is similar to the creation of Camera view, except that the method called at the end is different.

4.7 Application Running and Video Call

Once the remote user view is set up, you can theoretically now display your own and each other’s images and hear each other.

Running the program on two separate phones can verify the effect as follows:

In addition, you can also check the traffic usage of the current project on the official website:

10000 minutes of traffic per month, you can use it at ease

5. Summary of project development

With the above SDK access and code examples, a simple two-person audio and video call has been completed.

The Agora SDK offers far less.

We can also control the audio and video of the current channel, choose whether to mute, whether to close the picture and so on; We can also switch channels and participate in channel calls; We can also make the project a multi-person audio and video call, creating a remote view for each user joining the channel.

More audio and video related functions are available through the Agora SDK.

It is also convenient to summarize the overall access process:

SDK integration -> Permission Settings -> Camera preview -> Join channel -> Display screen -> Online call

Each of the above steps is explained in the article, I hope there will be some help in your use process!!

“This article was originally written by the Developer community of Sonnet.”

Details: rtcdeveloper.com/t/topic/200…

Audio net SDK access and audio and video call application development guide

1. Introduction

2. Account registration and project creation

3. SDK download and integration

4. Application development

4.1 Permission Application

4.2 Engine Creation

4.3 Realize Camera preview

4.3.1 Create Camera preview control View

4.3.2 Configure Camera output and enable preview

4.3.3 More configuration operations of Camera

4.4 Adding a Call Channel

4.5 Audio and video data encoding format parameter setting

4.5.1 Video Coding Parameter Settings

4.5.2 Audio Coding Parameter setting

4.6 User Joins and Displays the Peer screen

4.6.1 Determine whether a user joins a channel

4.6.2 Creating a Remote User View

4.7 Application Running and Video Call

5. Summary of project development

Related Posts

Weak network environment construction scheme selection

Gradle multichannel packaging (dynamically setting App names, App ICONS, replacing constants, changing package names, changing channels)

Signature analysis of a trendy e-commerce App newSign