WebRTC series
- WebRTC source research (1) WebRTC architecture
- WebRTC source code research (2) WebRTC source directory structure
- WebRTC source research (3) WebRTC operation mechanism
- WebRTC source research (4) Web server working principle and common protocol basis
- WebRTC source code research (5) Nodejs build environment
WebRTC source code research (2) WebRTC source directory structure
The last blog post, “WebRTC Architecture,” gave a rough overview of what WebRTC’s main modules can do. We know that WebRTC is an open source code acquired by Google with 60 million DOLLARS in 2010. Basically, there is no such bold and bold company in China. Therefore, to learn advanced technology, we still need to see foreign documents and open source community.
WebRTC is an audio and video communication treasure box, providing mature solutions for audio and video processing and instant messaging. The key is that the source code for this solution is open source, so you can delve into the source code and learn the algorithms for solving difficult problems and apply them to your projects. WebRTC is an excellent multimedia framework with cross-platform features.
WebRTC, a technology for real-time video and audio communications inside a browser, was acquired by Google through its $68.2 million acquisition of Global IT Solutions in 2010. Url is: code.google.com/p/webrtc/We… WebRTC uses iLBC iSAC, G711 and G722 voice protocols and VP8 video protocols. Implements a web-based video conferencing that enables developers to achieve real-time communication (RTC) capabilities by providing simple javascript through the browser. Developers can also access for WebRTC sites.google.com/site/webrtc page source code, specifications and tools, etc.
1. Download WebRTC source code
WebRTC source is very large, the current source for 10 g, if you don’t have a good network, it is very difficult to download, here I provide a source of stored in the baidu cloud, has the need to click here to download: WebRTC source download: link: pan.baidu.com/s/1BIJcRSzC… Password: 0 l8u
2. WebRTC source directory structure
2.1 WebRTC source directory summary
Because WebRTC is a very large, complex audio and video instant messaging technology open source library, the logic here is also very complex, if you do not know the directory structure, each directory is mainly what to do, if you want to query something is very difficult. So it’s worth taking a closer look at the directory structure.
- The main directory is roughly as follows:
| directory | function |
|---|---|
| api | WebRTC interface layer, through which the browser invokes WebRTC |
| call | At the management level of data flow, Call represents the inflow and outflow of all data from the same endpoint |
| video | Video related logic |
| audio | Audio related logic |
| common_audio | Audio algorithm correlation |
| common_video | Video algorithm correlation |
| media | Multimedia related logic processing, such as codec logic processing |
| logging | Logs related |
| module | The most important directory, submodules |
| pc | Peer Connection: the logical layer related to the Connection |
| p2p | End-to-end code, STUn,turn |
| rtc_base | Basic code, such as thread, lock related unified interface code |
| rtc_tool | Audio and video analysis related tools code |
| tool_webrtc | WebRTC tests related tools such as network emulators |
| system_wrappers | Code specific to the specific operating system. Such as CPU characteristics, atomic operations, etc |
| stats | Store all kinds of data statistics related classes |
| sdk | Store Android and IOS layer code. Such as video collection, rendering and so on |
Next on the above directory one by one brief description, specific or to see the source code:
-
The first is the API directory, which as the name implies, you can immediately guess, is the WebRTC interface layer, whether it is the browser, whether it is our own written applications, most of the time we have to call the API is directly from WebRTC through these API calls. But if I don’t want to add an interface, or if I want to change an interface I have to go to the API directory and find the corresponding file.
-
The second is call this directory, call this directory may be more difficult to understand, in fact is not very difficult, it is mainly the management of data streams, the audio stream and video when I connect the end, the same end the flow management is the management through the call, so, just set up a call such a directory, It represents the inflow and outflow of the same end. But if I’m talking to multiple ends, there are multiple calls.
-
The third one is video. Video is the logic related to video, video processing, video codec and so on, a general entry point.
-
The fourth one is audio, and audio is the logic and processing associated with audio.
-
The following two common_audio and commin_video, they’re algorithmic, so they’re algorithmic for audio and they’re in Common_audio, and the video is in Common_video. If you want to solve algorithmic problems, look in these two directories.
-
So then media,media is also very easy to understand, in fact, multimedia related processing, like the logic of codec, when to encode and when to decode, so these are in the media directory, so you have to solve the codec and decoding and what to do after decoding, You need to look it up in the media directory.
-
Next comes logging, which is log related. Generally, changes are few and simple.
-
The most important one here is the Module. The module is very big. There are many sub-modules in the module, and each sub-module is very important. The first thing we need to know is that the Module directory is particularly important.
-
Then there is PC. The meaning of PC is Peer Connection, which has been introduced to you before. Peer Connection represents a Connection, such as a Connection between me and the Peer end, which has the concept of Connection in the whole upper layer. That in this connection the following there are a lot of things, such as Stream flow, Track rail, a flow can contain many rail inside, including audio Track, and have desktop video rail tracks, etc., so the concept of orbital everyone should also understand, two parallel lines is the two orbitals, they are always disjoint, is such a concept, Audio and video never cross, they are stored separately. So through Peer Connection you can get streams, you can get every multimedia, that is, audio, video, you can also get the statistics of media streams, transmission statistics and so on, so Peer Connection is a very important directory. Is the upper equivalent of a unified interface layer.
-
Peer-to-peer transmission depends on whether p2p can get through. There are many types of P2P, which will be introduced later. Which type are you and whether this type can get through? The corresponding detection tools are put in the p2p directory, and the corresponding STUN protocol and turn protocol are also put in the P2P directory.
-
Rtc_base, which you can see at a glance, some basic things, such as the locks we use and the threads we use, are actually in, because it is cross-platform, it uses very different functions on different platforms, especially Windows and Linux. So how to do, when writing for my upper logic according to the platform to do processing, it unified to do a layer of directory.
-
Rtc_tool contains some tools, mainly audio and video analysis tools, such as I H264 for analysis, there is this I frame, P frame, B frame including some finer NL header can be found by rTC_tool corresponding tools.
-
There is also a tool called Tool_webrtc, which is a testing tool, and some unit tests, and various network tests, as well as audio and video tests, are all inside this tool_webrTC, and the main thing here is that the whole webrTC strategy is inside this tool_webrTC.
-
Next up is system_wrappers, which is a little more understandable, depending on the platform, your CPU, atomic operations, specific locks and so on, which are put in system_wrappers. Windows forms a file, Linux forms a file.
-
Stats is a class that is related to various statistics, such as my packet loss rate, jitter frequency, etc. This is a variety of statistics.
-
Below is the SDK mainly store Android and IOS code layer, including audio collection, audio rendering, are placed in the SDK directory.
-
The downloaded source folder directories are as follows:
├─./ Base // Basic Platform library, including Thread, lock, socket, etc. Gyp ├ ─ ─. / common_audio based public / / audio processing │ ├ ─ ─ the. / common_audio/include / / is a type conversion header file │ ├ ─ ─ the. / common_audio/resampler / / audio │ resampling code ├─./ Common_audio/Signal_Processing // With assembly code │ └ ─ ─. / common_audio/vad/code/vad ├ ─ ─. / common_video / / public video processing, Such as I420 frame processing, scaler, plane ├ ─ ─ the. / examples/examples / ├ ─ ─ the. / libjingle / / libjingle ├ ─ ─ the. / modules │ ├ ─ ─. / modules/audio_coding │ │ ├ ─ ─ /modules/audio_coding/codecs // audio coDEC processing, unified encapsulation of public interface and various different coDEC specific processing, │ ├─ │ ├─ /modules/ Audio_coding /main/acm2./modules/ Audio_coding /main/acm2 The main code / / audio processing module │ │ └ ─ ─ the. / modules/audio_coding/neteq/code/neteq │ │ ├ ─ ─. / modules/audio_coding/neteq/interface │ │ ├ ─ ─ ├── ├./modules/ exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises/exercises ├ ─ ─. / modules/audio_conference_mixer/code/audio synthesis │ │ ├ ─ ─ the. / modules/audio_conference_mixer/interface │ │ └ ─ ─ /modules/ Conference_Mixer /source │ ├─./modules/ Conference_Mixer/Source │ ├─./ Modules/Conference_Mixer/Source │ ├─. Win │ │ ├─ / Modules/Audio_Processing/Main │ ├─ /modules/ Audio_processing/Main // Audio_Devicemodule Aecm, agc, beamformer, ns, transient │ ├ ─ ─ the. / modules/bitrate_controller / / rate control module code │ ├ ─ ─ the. / modules/desktop_capture │ ├─./ / Modules /interface │ ├─./ /modules/media_file Support avi │ ├─./modules/pacing // bitrate detection code │ ├─./modules/remote_bitrate_estimator // remote bitrate calculation │ ├─./modules/rtp_rtcp │ ├─ feC │ ├─ /modules/ Utility │ ├─ /modules/ Video_capture Android, ios, Linux, MAC, Win │ ├─ /modules/ Video_coding // │ ├─ /modules/ Video_coding /codecs │ ├─ /modules/ Video_coding /main //VideoCodingModule ├─ /modules/ Video_coding /main //VideoCodingModule /modules/video_processing // Brighten color enhancement │ ├ ─ ├ ─ 07.07./ MB/MB/MB/MB/MB/MB/MB/MB/MB/MB/MB // Video rendering code, Android, ios, Linux, MAC, Windows, Opengles ├─./p2p // NAT pass code, turn/stun etc. Server and client │ ├ ─ ─. / p2p/base │ └ ─ ─ the. / p2p client ├ ─ ─ the. / sound / / unknown ├ ─ ─ the. / system_wrappers / / system API encapsulation ├ ─ ─. / test ├ ─ ─. / tools / / audio and video test tools code ├ ─ ─ the. / video / / unknown ├ ─ ─. / video_engine/code/video engine, the video processing └ ─ ─. / voice_engine/code/audio engine, audio processingCopy the code2.2 WebRTC source directory Function description
With a general explanation of directories above, let’s look at the specific source file directory information from the command line
2.2.1 WebRTC source directory
| directory | function | The path | instructions |
|---|---|---|---|
| api | WebRTC interface layer, through which the browser invoks WebRTC, including DataChannel, MediaStream, and SDP-related interfaces | ||
| call | At the management level of data flow, Call represents the inflow and outflow of all data at the same point and stores the code of WebRTC’s “Call” related logic layer. | ||
| video | Video related logic, video logic layer and video engine layer related code. Video data logical send, receive and other code. | ||
| audio | Audio related logic, audio data logic on the send, receive code. | ||
| common_audio | Audio algorithm related, store some basic audio algorithm. Including ring queue, Bollie algorithm, filter and so on. | ||
| common_video | Video algorithms, including libyuv, SPS/PPS analyzer, I420 buffer, etc. | ||
| media | Multimedia related logic processing, such as codec logic processing | ||
| logging | Logs related | ||
| modules | The most important directory, submodules | ||
| pc | Peer Connection Stores the code of the service logic layer. Such as the channel, the session, etc | ||
| p2p | End-to-end correlation code, STUN, turn | ||
| rtc_base | Basic code, such as thread, lock related unified interface code | ||
| rtc_tools | Audio and video analysis related tools code | ||
| tools_webrtc | WebRTC tests related tool code, such as network emulator | ||
| system_wrappers | Code specific to the specific operating system. Such as CPU characteristics, atomic operations, etc | ||
| stats | Store all kinds of data statistics related classes | ||
| sdk | Store Android and IOS layer code. Such as video collection, rendering and so on | ||
| base | |||
| build | |||
| build_overrides | |||
| buildtools | |||
| crypto | |||
| data | |||
| examples | |||
| ios | |||
| media | |||
| out | |||
| resources | |||
| style-guide | |||
| test | |||
| testing | |||
| third_party | |||
| tools | |||
2.2.1.1 API directory
WebRTC interface layer, through which the browser invoks WebRTC, including DataChannel, MediaStream, and SDP-related interfaces
2.2.1.2 call directory
At the management level of data flow, Call represents the inflow and outflow of all data at the same point and stores the code of WebRTC’s “Call” related logic layer
2.2.1.3 video directory
Video related logic, video logic layer and video engine layer related code. Video data logical send, receive and other code.
2.2.1.4 audio directory
Audio related logic, audio data logic on the send, receive code.
2.2.1.5 common_audio directory
Audio algorithm related, store some basic audio algorithm. Including ring queue, Bollie algorithm, filter and so on.
2.2.1.6 common_video directory
Video algorithms, including libyuv, SPS/PPS analyzer, I420 buffer, etc.
2.2.1.7 media directory
Multimedia related logic processing, such as codec logic processing
- Media/base directory
- Media/engine directory
- Media/SCTP
2.2.1.8 logging directory
Logs related
2.2.1.9 modules directory
Modules The most important directory, the submodule, is introduced here:
| directory | function | note |
|---|---|---|
| audio_coding | Audio codec related code | Audio_coding, which is related to codec logic, will use codecs, so audio_coding in this case is the codec, it’s all in the audio_coding directory. |
| audio_device | Video capture and audio playback related code | Audio_device is devicerelated, it makes a distinction, Android and IOS are under the SDK, some of its code is moved into the SDK, in previous versions of WebrTC, all devices are in the Audio_device, Both our Windows and MAC and Linux are in the audio_device directory. |
| audio_mixer | Remix related code | An audio_mixer is related to mixing. Mixing is similar to a meeting in which multiple people interact in real time. For example, four people speaking at the same time are mixed together to facilitate transmission, which reduces the flow of audio. |
| audio_processing | Audio pre – and post-processing related code | Audio_processing is the pre-processing and post-processing of audio. The pre-processing and post-processing of audio, such as echo cancellation, noise reduction, and gain, are put in audio_processing. There are a number of subdirectories in this directory, and you’ll get a general idea of the related ones in this directory later. And find subdirectories under this directory |
| bitrate_controller | Bit rate control related code | Bitrate_controller, this is the bit stream control, for example, controlling the bit stream, whether the video bit stream is 500K or 1M. This bit stream control. If you want to know how he controls the relevant bit stream, you can go here. |
| congestion_controller | Flow control related code | – Congestion_controller is traffic control. When I can detect high traffic on the network, I need to do some traffic control to prevent the network packet from killing bandwidth. So if I want to know anything about flow control, I go to this directory. |
| desktop_capture | Desktop collection related code | — |
| pacing | Bit rate detection and smoothing related code | The function of pacing is bit rate detection and smoothing, I first detect what the bit rate of audio and video is, then we need to do a smoothing process, we can not let it send out all of a sudden, we should make it smooth, for example, sometimes it is 10K, sometimes it is 500K, we let him average. |
| remote_bitrate_estimator | Code related to remote bit rate estimation | The emote_bitrate_estimator is used to evaluate the remote bitrate. What is the bandwidth that I can receive remotely? One is my local bandwidth, and the other is the remote bandwidth. This is put in the remote_bitrate_estimator directory |
| rtc_rtcp | RTP/RTCP related codes | — |
| video_capture | Video capture related code | Video_capture I capture the relevant code of the video and put it in video_capture |
| video_coding | Video codec related codes | Video encoding, such as H264,VP 8,VP 9, etc., the encoder is put in video_coding. |
| video_processing | Handle the code before and after the video | Video_processing is the pre-processing and post-processing of video, such as video frame enhancement, detection, if we add face recognition can be put under the video_processing directory. |
Next, I’ll use the command line to check:
- Modules/audio_coding directory
- Modules/audio_device directory
- Modules/audio_mixer directory
- Modules/audio_processing directory
- Modules/bitrate_controller directory
- Modules/congestion_controller directory
- Modules/desktop_capture directory
- Modules/include directory
- Modules/pacing
- Modules/remote_bitrate_estimator directory
- Modules/rtp_rtcp directory
- Modules/third_party directory
- Modules/utility
- Modules/video_capture directory
- Modules/video_coding directory
- Modules/video_processing directory
2.2.1.10 PC directory
Peer Connection Stores the code of the service logic layer. Such as the channel, the session, etc
2.2.1.11 p2p directory
End-to-end correlation code, STUN, turn
2.2.1.12 rtc_base directory
Basic code, such as thread, lock related unified interface code
2.2.1.13 rtc_tools directory
Audio and video analysis related tools code
2.2.1.14 tools_webrtc directory
WebRTC tests related tool code, such as network emulator
2.2.1.15 system_wrappers directory
Code specific to the specific operating system. Such as CPU characteristics, atomic operations, etc
2.2.1.16 system_wrappers directory
2.2.1.17 stats directory
Store all kinds of data statistics related classes
2.2.1.18 SDK directory
Store Android and IOS layer code. Such as video collection, rendering and so on
2.2.1.19 base directory
2.2.1.20 build directory
2.2.1.21 build_overrides directory
2.2.1.22 buildtools directory
2.2.1.23 crypto directory
2.2.1.24 data directory
2.2.1.25 examples directory
2.2.1.26 ios directory
2.2.1.27 media directory
2.2.1.28 out directory
2.2.1.29 resources directory
2.2.1.30 style – guide catalog
2.2.1.31 test directory
2.2.1.32 testing directory
2.2.1.33 third_party directory
2.2.1.34 tools directory
2.2.2 WebRTC Modules subdirectory
The Modules module is the most important directory in WebRTC source code
| directory | function | The path | instructions |
|---|---|---|---|
| audio_coding | Audio codec related code | ||
| audio_device | Audio acquisition and audio playback related code | ||
| audio_mixer | Remix related code | ||
| audio_processing | Audio pre – and post-processing related code | ||
| bitrate_controller | Bit rate control related code | ||
| congestion_controller | Flow control related code | ||
| desktop_capture | Desktop collection related code | ||
| pacing | Bit rate detection and smoothing related code | ||
| remote_bitrate_estimator | Code related to remote bit rate estimation | ||
| rtp_rtcp | RTP/RTCP related codes | ||
| video_capture | Video capture related code | ||
| video_coding | Video coding and decoding related codes | ||
| video_processing | Handle the code before and after the video | ||