The first AI in RTC Competition, organized by Agora, was successfully concluded on October 24. Among them, more than 100 teams participated in the creative programming track and brought many projects with their creativity and enthusiasm in more than three months. We invited the winning teams to share their development practices one by one.

Buggy’s team developed a video-enabled car that can be remotely controlled from a mobile phone based on Android phones, development boards, and sensor components.

The following was written by the Buggy team:

With the rapid development trend of consumer electronic devices, the price of smart phones has become lower but the configuration has become higher. More and more people choose to replace smart phones with higher configurations. If old mobile phones are not recycled, the heavy metals and plastics in them will cause serious and irreversible pollution to the environment. Unfortunately, most old smart phones still work normally. At present, the mainstream way to dispose of old mobile phones is to recycle and refine precious metals, but the process can still produce toxic and harmful gases.

Smart phone cameras already meet the general requirements for image acquisition. Using old smart phone cameras to develop video monitoring applications has become a major trend of reuse of old phones [1], which can save a lot of cost of video acquisition equipment and reduce environmental pollution and resource waste.

Rico, a smart security product developed by the start-up team of Mindhelix in India, uses old smartphones to create an innovative security product. Rico is mainly composed of hardware and software such as old smart phones, smart sockets, various environmental sensors and mobile applications built into Rico. Load an old smartphone with an app and plug it into Rico, and the otherwise scrapped smartphone takes over video cameras, facial recognition, voice control, microphone calls, trip reminders and music playback. Rico’s built-in environmental sensors monitor carbon monoxide, gas, smoke concentrations, temperature, humidity, light intensity and movement; Smart sockets can be connected to a variety of household appliances and lights in the home and control them on and off; All the monitored data is uploaded to the cloud through THE API interface, and the indicators of the home environment can be viewed on the mobile application. Such as temperature, humidity, etc. If you forget to turn off the light when you go out, you can turn off the light remotely through your mobile phone [2].

Alfred, which is popular in North America, is an APP developed by a start-up team in Taipei that turns old mobile phones into surveillance cameras, offering motion detection, push notifications, real-time video recording and night vision. Users only need to install apps on two smart phones, one set as camera terminal and the other as monitoring terminal, and log into the same Google account at both ends to achieve remote monitoring through WIFI or 3G/4G [3][4].

Research ideas

Nowadays, in the field of security monitoring, most civil monitoring equipment has several disadvantages:

  1. Fixed position: most devices are fixed in position and inconvenient to move. Even if some devices can rotate the camera, they still cannot monitor most areas. For example, some dead corners will not be monitored (bed bottom, corner, etc.).
  2. Wired transmission: Most are wired, and even some are wireless, but still tied to the power cord.
  3. Expensive: A complete set of monitoring equipment costs more today.

Thought planning and work design

Concept planning

This design requires that a mobile phone can remotely view the picture taken by the mobile phone camera on the car through WIFI and control the car’s movement. The camera of obsolete smart phone is used to collect video image signal in real time and transmit the video signal to the mobile phone end for watching through wireless WIFI network. Then the infrared remote control function of the mobile phone on the car is used to control the moving direction and speed of the car. Use C language to develop microcontroller program, use Java language to develop Android program.

This section discusses the selection of Android video transmission scheme, single chip microcomputer, car body, motor drive and remote control module.

Select the video transmission scheme of Android terminal

After consulting relevant literature and materials, the image data processing methods of Android real-time image transmission are summarized, and five Android real-time image network transmission schemes are summarized in the following table [5].

In the Internet environment, option 1 is not desirable. Scheme 2 MediaRecorder has a strong hardware dependence, so different mobile phone performance is not the same, some mobile phone data transmission will appear blocking phenomenon, real-time is reduced. Scheme 3 and Scheme 4 have low real-time performance. Option 5, with its high average traffic consumption, is also undesirable.

Another scheme is adopted here, which uses Agora’s video call SDK for secondary development. It not only ensures real-time performance, but also reduces the difficulty of development.

MCU choice: MCU is used to receive infrared signal and control motor drive, according to the online tutorial more MCU is STC89CXX series, STC89C52RC MCU is the upgrade version of STC89C52, the continuous development of semiconductor technology also let the MCU power consumption is constantly reduced. It not only achieves the upgrade of energy consumption, but also further strengthens the processing performance of single chip microcomputer. Therefore, STC89C52RC model is selected.

Car body selection: The first option is to purchase the car body for this design on Taobao. The chassis and top surface of the car adopts 3mm thick acrylic plate, which is formed by laser cutting, and a plurality of mounting holes are reserved, with strong expansion ability. Using the characteristics of the car’s round chassis, dc motors and rubber anti-skid wheels on the left and right sides of the bottom surface, and two universal wheels on the front and rear play the role of balancing the car. Dc motor is easy to control and speed, and this kind of toy frame is generally cheap; The second option is to make a car body. Made of aluminum alloy plate, cutting plate drilling is required. This process is time-consuming and should be better spent on program development. So use the first plan to buy the body from Taobao.

Selection of motor driver: Scheme 1, control motor motion state through relay. The advantage is that the structure is simple, but the working principle is mechanical structure, so the disadvantage is slow reaction, and occupy a large space; Scheme 2: L293D, the motor driver chip of TI company, is adopted. Maximum voltage 36V, maximum peak current 1.2a, 0.6A continuous rated current. Use TTL level signal control. It has two enable controllers, which can be used as PWM signal input to adjust motor speed [5]. To sum up, plan 2 is selected.

Remote control module selection: Scheme 1: infrared remote control. Although the control distance of the infrared remote control is only about 10m, it cannot be controlled around obstacles. But the mobile phone that emits infrared remote control signal is on the trolley, and the infrared transmitter and infrared receiver of the mobile phone can be fixed together. Although not all Android phones have infrared emitters, they all have 3.5mm headphone jack, and the infrared signal frequency of 38kHz is within the audio range. Infrared remote control signals can be transmitted by the infrared light-emitting diode connected to the headphone jack [6]. In scheme 2, bluetooth can be used to control the car, and the car needs to be equipped with bluetooth module for pairing communication with the mobile phone. But not all phones support Bluetooth, and some early Android smartphones didn’t. Infrared modules are cheaper than Bluetooth modules. So plan 1 is adopted.

Design concept

This design is based on android phone as the control core, in the mobile phone control terminal real-time check the image collected by the mobile phone camera on the car, and send control commands to the mobile phone on the car, and then send action instructions through the infrared control terminal of the mobile phone. VS1838B is the infrared receiving module, receiving the infrared remote control signal of the mobile phone end, and then handing it to THE STC89C52RC microcontroller to judge the code value of the infrared remote control, corresponding to the control L293D motor drive, so as to achieve the car forward, forward and left. The power supply adopts two 18650 lithium batteries in series.

3.4 System Block Diagram The system block diagram of the system is shown in figure.

Project implementation

Car hardware circuit design

The whole car hardware design can be divided into four modules: power supply circuit, SCM minimum system, motor driving module, infrared receiving module. The power supply circuit supplies power to the minimum system of single chip microcomputer, infrared receiving module, motor drive and two wheel motors.

Microcontroller minimum system

The minimum system of the single chip microcomputer is a necessary part for the single chip microcomputer to work normally and play its functions, and can also be understood as a system that the single chip microcomputer can work with the least number of components [7]. The system block diagram and circuit schematic diagram of the minimum SCM system are shown in the figure.

STC89C52RC microcontroller is a milestone in the development of domestic microcontroller, initially achieved domestic mass production. The 52 microcontroller is an upgraded version of 51, and the continuous development of semiconductor technology also makes the power consumption of the microcontroller constantly reduced. It not only achieves the upgrade of energy consumption, but also further strengthens the processing performance of single chip microcomputer. Under the principle of keeping the original 8-bit processor unchanged, the PRODUCTION process of CMOS STC89S52RC MCU is integrated with 512 bytes RAM and 8K bytes ROM. In the burn times of about 1000 times of excellent performance, low power mode of the clock system will stop working, but the internal register can still be normal processing. In order to adapt to different use occasions, this single chip microcomputer provides a variety of packages, the design according to the minimum system sometimes need to replace the specific situation of the single chip microcomputer, the use of double in-line diP-40 package, with a compact base.

Motor drive module

The L293D is a TI product and is commonly available in a 16-pin PDIP package. L293D is controlled by TTL level signal, driven by OUTPUT1, OUTPUT2 and OUTPUT3, OUTPUT4. INPUT1, INPUT2, INPUT3, INPUT4 pins are connected to ports P0, P0.1, P0.2, p0.3 of STC89C52RC to control the motor motion state. ENABLE1, ENABLE2 is the control enabler, and is connected to port P0.4 and P0.5 of the single chip microcomputer to control the stopping of the motor and PWM speed regulation [8]. P0.4 and P0.5 ports change the speed of the motor by output PWM waveforms with different duty cycles. Duty cycle is the percentage of high level duration over a cycle time. Single chip microcomputer output PWM signal can use software delay or timer two methods, in this is the software delay method. When the high level delay time arrives, the level inverts to the low level, and then delays; When the low level delay time arrives, the level is inversely changed to high level, and the PWM waveform can be obtained by doing this repeatedly [9]. VSS can be connected to 4.5 ~ 7V voltage to provide voltage for the logic circuit of the chip. The VS voltage range is 4.5 to 36 V, providing voltage to the output terminal. Table 4-1 describes the mapping between the truth logic and functions of L293D. Table L293D truth logic and function corresponding table

Infrared remote control module

Infrared remote control module is mainly composed of remote control transmitter, infrared receiving head and interface circuit. Infrared remote control is used to generate remote control coding pulse, drive infrared transmitting tube output infrared remote control signal, the use of the mobile phone with infrared emission function. Remote control receiving head to complete the remote control signal amplification, detection, shaping, demodulation of the remote control code pulse, the use of integrated infrared receiving head to complete this series of operations. Remote control coding pulse is a set of serial binary code, which is input to the single chip microcomputer, and its internal CPU decodes the remote control instruction and performs the corresponding remote control function [9].

Realization principle of infrared remote control

This infrared remote control adopts NEC protocol, modulation mode is PPM pulse position modulation, carrier 38kHz. The difference between bit 0 and bit 1 is that the pulse width of low level after high level pulse is different. Logic 1 is 2.25ms and pulse time is 560us. Logic 0 is 1.125ms, pulse time 560us. Its waveform is shown in figure.

Below is the format of the NEC protocol. The first is the boot code, the high level pulse of 9ms and the low level of 4.5ms, followed by the 8-bit address code and the 8-bit address inverse code to verify the address code, followed by the 8-bit command code and the 8-bit command inverse code to verify the address code [10].

Infrared receiving module

VS1838B integrated infrared receiver has three pins, as shown in Figure 4-6. OUT end is the output end of demodulation signal, which is connected with port P3-2 of SCM.

Car software Design

This design can be programmed by C language or assembly language. As the internal capacity of SCM is relatively large, programs can be programmed by C language without the use of obscure assembly language [11]. So the system program in this design uses C language to write programs, compiled with software Keil C51.

Main programming

In this design, the single chip microcomputer is mainly under the control of the main program, the infrared signal received by the infrared integrated receiving head VS1838B is identified and analyzed regularly. If the data read out are 0x12, 0x18, 0x14, 0x16, it corresponds to the forward, backward, left, right turn of the control car respectively. Figure 4-7 shows the flow chart of hardware system software design.

The main code is as follows.

void main(){ IrInit(); // initialize ENA=0; ENB=0; T=2048; speed=400; // The higher the number, the faster the speed. ENB=1;while(1){ detectorIR(); MotorRun (); // Control the motor stateif(speed! =T){ ENA=0; ENB=0; delay(T-speed); ENA=1; ENB=1; delay(speed); }}}Copy the code

Infrared remote control receiving program design

The center of gravity of infrared remote control receiving program lies in the way of decoding, decoding method is as follows:

  • Step 1 Set the external interrupt 0(or 1) as the falling edge interrupt, and define a variable Time to be used for timing. The initial value is 0.
  • Step 2 Start the timer after the remote control is interrupted for the first time. From the second time to enter the remote control interruption, first stop timing. And save the timing value, and then re-time. The lead flag is true if the timed value is equal to the time of the lead. Prepare to receive the following frame of remote control data. If the timing value is not equal to the time of the lead code, but the lead code has been received before, judge whether it is 0 or 1 of the remote control data.
  • Step 3 Continue to receive the address code, address inverse code, data code, and data inverse code.
  • Step 4 When 32-bit data is received, a frame is received. If the two address codes are the same and equal to the address of the system, and the sum of the data code and the data inverse code is equal to OFFH, the received frame data is valid. Otherwise, the received data is discarded.
  • Step 5 Save the received data and prepare for the next remote reception.

The main code is as follows.

void ReadIr() interrupt 0
	{
		u8 j,k;
		u16 err;
		Time=0;					 
		delay(700);	//7ms
		if(IRIN==0) // Verify that the correct signal is actually received {err=1000; //1000*10us=10ms, if both conditions are true, break out of the loop if either condition is false, so that the program will not die if it fails */while((IRIN==0)&&(err>0)) {delay(1); err--; }if(IRIN==1) // If correct wait until 9ms low {err=500;while((IRIN highlights = = 1) && (err) > 0) / / wait for 4.5 ms initial high level past {delay (1); err--; }for(k=0; k<4; K++) // there are 4 groups of data {for(j=0; j<8; J++) // receive a set of data {err=60;while((IRIN highlights = = 0) && (err) > 0) / / waiting for the signal in front of us low level over the past 560 {delay (1); err--; } err=500;while((IRIN==1)&&(err>0)) // The length of time to calculate the high level. { delay(10); / / 0.1 ms Time++; err--;if(Time>30)
							{
								return; } } IrValue[k]>>=1; //k indicates the number of rowsif(Time > = 8) / / if high level in more than 565 us, it's 1 {IrValue [k] | = 0 x80; } Time=0; // Use up the time to reassign}}}if(IrValue[2]! =~IrValue[3]) {return; }}}Copy the code

Android Programming

The Android program of this design can use the camera of the mobile phone to transmit the image captured by the mobile phone to the mobile phone or computer terminal in real time, and receive the instructions from the mobile phone or computer terminal. According to the instructions, the mobile phone on the car will send the corresponding infrared remote control signal to the car. The mobile phone Android program is the car end and the control end of the two in-one, select in the initial interface, mainly to achieve the reception and transmission of signaling, real-time audio and video transmission and infrared signal transmission and other functions.

This development uses Agora video call and signaling SDK for secondary development. The following describes the development environment to be prepared before using Agora SDK for Android for call/live broadcast and signaling control, including the prerequisites and SDK integration methods [12]. Ensure that the following development environment requirements are met:

  1. Android Studio 3.0 or later
  2. App requires Android 4.1 or above
  3. Android SDK API Level ≥16

Before compiling and starting the instance program, you need to first get an available App ID:

  1. Create a developer account in agora. IO
  2. Go to the background page, click “project” → “Project List” menu in the left navigation bar
  3. Copy the App ID in the background and note that it will be used later when launching the App

Integrate Agora video SDK:

  1. Implementation ‘IO. Agora. RTC :full-sdk:2.4.1′; implementation’ IO. Agora. RTC :full-sdk:2.4.1′;
  2. Download agora-RtC-sdk. jar and agora-sig-sdk.jar from the official website of agora. IO. Copy all header files in the libs/include folder to the app/ SRC /main/ CPP/Agora directory.

Once you’ve completed the above steps, open the project with Android Studio, connect the device, and compile and run the application.

The design of mobile terminal program

The ConsumerIrManager class is used to control infrared devices only on Android4.4 and above, so this program is developed and tested based on Android 5.1 OPPO A51 mobile phone. Developed with Android Studio, it is very convenient to complete relevant application development of Android mobile terminals in this development environment [13]. The program flow chart of real-time video function of mobile phone control terminal is shown in the figure below.

Please refer to the reference [12] on the official website of Sonnet for the specific usage method, which will not be described in detail here. Here is a brief introduction to the main steps.

(1) the initialization

Before entering the channel, call create to create an instance.

import io.agora.rtc.Constants; import io.agora.rtc.IRtcEngineEventHandler; import io.agora.rtc.RtcEngine; import io.agora.rtc.video.VideoCanvas; // Import these 4 packages before initialization... private voidinitializeAgoraEngine() {
    try {
        mRtcEngine = RtcEngine.create(getBaseContext(),
getString(R.string.agora_app_id),
                              mRtcEventHandler);
    } catch (Exception e) {
        Log.e(LOG_TAG, Log.getStackTraceString(e));
        throw new RuntimeException("NEED TO check rtc sdk init fatal error\n"+ Log.getStackTraceString(e)); }}Copy the code

(2) Join the channel

Before joining the channel, the App needs to set the channel mode and then join the channel. After creating the instance, call the setChannelProfile method to set the channel mode.

mRtcEngine.setChannelProfile(Constants.CHANNEL_PROFILE_COMMUNICATION);
Copy the code

Then call the joinChannel method to join the channel.

 private void joinChannel() {
    mRtcEngine.joinChannel("token"."demoChannel1"."Extra Optional Data", 0); 
}
Copy the code

(3) Leave the channel

Call the leaveChannel method to leave the channel and end or exit the call or live broadcast.

 private void leaveChannel() {
    mRtcEngine.leaveChannel();
}
Copy the code

(4) Signaling implementation

The specific use method of signaling can be found in the reference [14] of the document center on sonnet’s official website, without detailed description here. The main functions used are as follows:

/ / initialize the signaling SDK m_agoraAPI = AgoraAPIOnlySignal. GetInstance (context, appID); M_agoraapi. login2(appId, Account, token, uid, deviceID, retry_time_in_s, Retry_count) / / / / / / / / / / / / / channel test / / / / / / / / / / / / / / / / / join channel m_agoraAPI channelJoin (channelName) / / channel to send message M_agoraAPI. MessageChannelSend (channelName, MSG, msgID) / / set the channel to end receives the message callback m_agoraAPI. OnMessageChannelReceive (channelID, Logout () {// code there}Copy the code

The program flow chart of the real-time video function of mobile phone on the car is shown in the figure below.

The program flow chart of infrared emission function of mobile terminal is shown in the figure.

The ConsumerIrManager class is used to control infrared devices only on Android4.4 and above, so the following program is developed and tested based on Android 5.1 OPPO A51 phone.

The ConsumerIrManager service is first retrieved from the system service.

IR=(ConsumerIrManager)getSystemService(CONSUMER_IR_SERVICE);
Copy the code

The infrared code to be sent is then stored in the array.

//0x73 int[] pattern2 = {9000, 4500, 560,560, 560,560, 560,560, 560,560, 560,560, 560, 560,560, 560,560, 560,560, 560,560, 560,560, 560,560, 560,560, 560,560, 560,560, 560,560, 560, 1690, 560, 560, 560, 560, 560, 560, 560, 560, 560, 1690, 560, 560, 1690, 560, 560, 560, 560, 560, 1690, 560, 1690,560, 1690,560, 42020, 9000, 2250, 560, 98190};Copy the code

And what’s stored in the array is a number that represents an alternating carrier sequence pattern, measured in milliseconds. These six lines of data represent the guide code, address code, address code, data code, data inverse code, and continuous code in sequence. For details, see Section 4.1.3 Infrared Remote Control Principles.

The third line of data code is inverted, for example, 0x12=0001 0010 is inverted to 0100 1000. The data code can be received normally only after inverting. Finally, infrared signal is sent through the following methods.

mCIR.transmit(hz, pattern2); / / backCopy the code

Transmit (int carrierFrequency, int[] pattern) : This method controls the mobile phone to generate carrierFrequency and send infrared signals with pattern as the time array of infrared switch. (for example, transmit(38000,{100,200,300,400}) will generate an infrared signal with a frequency of 38KHz. The level of the signal is 100us high level, 200us low level, 300us high level, 400us low level. Note that the value of pattern must be an even number, otherwise an error is reported. [15].

References:

  • [1] ZHAO Yunle. Design and implementation of Video surveillance platform based on Android mobile camera [J]. Computer Programming Skills and Maintenance,2016(17):36-37.]
  • [2] Rice in shrimp shell. Mindhelix waste mobile phones into part of the smart home [OL]. www.shejipi.com/31209.html,… .
  • Recommended by [3] Chony Lu. Use Google App, make spare cell phone camera monitoring – Alfred phooey butler # iOS# Android [OL]. Chuansongme.com/n/145401142… .
  • [4] WANG Ye. Design of intelligent navigation Car based on Android System [D]. Tianjin: Tianjin University,2014.
  • [5] android real-time video network transmission plan summary (a total of five sets) [OL]. Blog.csdn.net/findsafety/… .
  • [6] android taught you how to study and implement the android headphones mouth turn audio infrared emission [OL]. Blog.csdn.net/u012534831/… .
  • [7] NIE Ru. Design and Implementation of Smart Car Based on Android Bluetooth Control [J]. Microcomputer applications,2015,31(09):68-69+74+6.
  • [8] ZHU Tao. Design of intelligent tracking car based on STC89C52 Microcontroller [J]. Computer knowledge and technology,2011,7(31):7751-7753+7758.
  • [9] ZHAO Hai-lan. Design of Infrared Remote Control Smart Car Based on Microcontroller [J]. Wireless Interconnection Technology,2011(03):36-38.
  • [10] Liu Xijiang. Realization of AGV Car Infrared Remote Control Based on SCM [A]. Henan Automotive Engineering Society. Proceedings of the 13th Henan Automobile Engineering Science and Technology Symposium [C]. Henan Automobile Engineering Society: Henan Automobile Engineering Society,2016:4.
  • [11] Xu Aijun. Microcontroller C language programming and Proteus Simulation Technology [M]. Beijing: Publishing House of Electronics Industry,2016.
  • The client [OL]. [12] integration docs. Agora. IO/cn/Video/an… .
  • [13] Peak excellence. Android from entry to Mastery [M]. Beijing: Posts and Telecommunications Press,2016.
  • [OL]. [14] channel messages sent docs. Agora. IO/cn/Signalin… .
  • [15]Android Programming – Infrared code detailed Analysis [OL].blog.csdn.net/sir_zeng/ar… .
  • [16] Song Jichao. Design and Implementation of Security Robot Control System [D]. University of Electronic Science and Technology of China,2018.
  • [17]Android Practice: A video interactive smart car [OL].blog.csdn.net/weixin_3378… .

You can visit team Buggy’s blog and talk to them. In the meantime, welcome to the RTC developer community.