Technical background

Whether it is Windows platform or Linux, the demand for multiplexing is very common, such as camera display in smart construction sites, pavilions, education and other macro scenes, regarding the RTSP or RTMP live player development points that need to be paid attention to, please refer to the previous blog, in general, there are some points as follows:

1. Low latency: Most RTSP play is for live scenes, so if the delay is too large, such as in the surveillance industry, the thief is gone, or someone has rung the doorbell for a few seconds, the owner will see the image, seriously affect the experience, so low latency is a very important indicator of a good RTSP player. At present, the RTSP playback delay of Daniu Live SDK is controlled in hundreds of milliseconds, and the VLC in a few seconds. This delay is a long low delay, such as 1 day, a week, a month or even longer;

* * 2. Audio and video synchronization or jump: * * some developers in pursuit of low latency experience, don’t even do the audio and video synchronization, to get the audio video playback, directly led to the a/v out of sync, and the timestamp disorderly jump.

**3. Support multi-instance: ** A good player, need to support the simultaneous playback of multi-channel audio and video data, such as 4-8-9-16-32 window;

4. Support buffer time setting: In some scenarios with network jitter, the player needs to support accurate buffer time setting. Generally, it is measured in milliseconds.

**5. H.265 playback and recording: ** in addition to H.264, also need to support H.265, RTSP camera on the market is more and more, support H.265 RTSP player imminent, in addition, simple playback OF H.265 is not enough, but also need to be able to record the DATA of H.265;

**6. TCP/UDP mode switching: ** Considering that many servers only support TCP or UDP mode, a good RTSP player needs to support TCP/UDP mode switching automatically;

**7. Mute support: ** for example, multi-window playback RTSP stream, if every audio is played out, the experience is very bad, so real-time mute function is very necessary;

**8. Video View rotation: ** Many cameras due to installation restrictions, resulting in image inversion, so a good RTSP player should support such as video view real-time rotation (0° 90° 180° 270°), horizontal inversion, vertical inversion;

**9. Support audio/video data output after decoding (optional) : ** SDK contact a lot of developers, hope to be able to play at the same time, get YUV or RGB data, face matching algorithm analysis, so audio and video callback is optional;

**10. Snapshot: ** Interested or important pictures, real-time intercession is very necessary;

**11. Network jitter processing (such as disconnection) : ** Basic functions, no further description;

**12. Cross-platform: ** A good player, cross-platform (Windows/Android/iOS) is necessary, at least for the subsequent scalability consideration, development, have this consideration, the current Bull live SDK RTSP player, perfectly support the above platforms;

13. Long-term stability: When it comes to stability, many developers don’t think much of it. In fact, a good product, stability is the most basic premise, can not be ignored! **14. Video recording: ** In the process of playing, record interested video clips at any time, archive or other secondary processing;

**15. Log information recording: ** the overall process mechanism real-time feedback, not many log, but not some important log, such as error in the process of playing;

**16. Download speed feedback: ** can see the real-time download speed feedback, so as to monitor the network status;

**17. Abnormal status processing: ** Such as in the process of playing, network disconnection, network jitter, telephone, cut back after the background and other scenarios processing.

Code implementation

This article takes the Linux platform of Daniu Live SDK (official) as an example to introduce the integration of RTMP or RTSP stream multiplexer.

int main(int argc, char *argv[]) { XInitThreads(); // NT_SDKLogInit() must be called; // SDK initializes SmartPlayerSDKAPI player_API; if (! NT_PlayerSDKInit(player_api)) { fprintf(stderr, "SDK init failed.\n"); return 0; } auto display = XOpenDisplay(nullptr); if (! display) { fprintf(stderr, "Cannot connect to X server\n"); player_api.UnInit(); return 0; } auto screen = DefaultScreen(display); auto root = XRootWindow(display, screen); XWindowAttributes root_win_att; if (! XGetWindowAttributes(display, root, &root_win_att)) { fprintf(stderr, "Get Root window attri failed\n"); player_api.UnInit(); XCloseDisplay(display); return 0; } if (root_win_att.width < 100 || root_win_att.height < 100) { fprintf(stderr, "Root window size error.\n"); player_api.UnInit(); XCloseDisplay(display); return 0; } fprintf(stdout, "Root Window Size:%d*%d\n", root_win_att.width, root_win_att.height); int main_w = root_win_att.width / 2, main_h = root_win_att.height/2; auto black_pixel = BlackPixel(display, screen); auto white_pixel = WhitePixel(display, screen); auto main_wid = XCreateSimpleWindow(display, root, 0, 0, main_w, main_h, 0, white_pixel, black_pixel); if (! main_wid) { player_api.UnInit(); XCloseDisplay(display); fprintf(stderr, "Cannot create main windows\n"); return 0; } XSelectInput(display, main_wid, StructureNotifyMask | KeyPressMask); XMapWindow(display, main_wid); XStoreName(display, main_wid, win_base_title); std::vector<std::shared_ptr<NT_PlayerSDKWrapper> > players; for (auto url: players_url_) { auto i = std::make_shared<NT_PlayerSDKWrapper>(&player_api); i->SetDisplay(display); i->SetScreen(screen); i->SetURL(url); players.push_back(i); if ( players.size() > 3 ) break; } auto border_w = 2; std::vector<NT_LayoutRect> layout_rects; SubWindowsLayout(main_w, main_h, border_w, static_cast<int>(players.size()), layout_rects); for (auto i = 0; i < static_cast<int>(players.size()); ++i) { assert(players[i]); players[i]->SetWindow(CreateSubWindow(display, screen, main_wid, layout_rects[i], border_w)); } for (const auto& i : players) { assert(i); if (i->GetWindow()) XMapWindow(display, i->GetWindow()); } for (auto i = 0; i < static_cast<int>(players.size()); ++i) { assert(players[i]); // players[I]->Start(0, I! =0, 1, false); //players[i]->Start(0, false, 1, false); } while (true) { while (MY_X11_Pending(display, 10)) { XEvent xev; memset(&xev, 0, sizeof(xev)); XNextEvent(display, &xev); if (xev.type == ConfigureNotify) { if (xev.xconfigure.window == main_wid) { if (xev.xconfigure.width ! = main_w || xev.xconfigure.height ! = main_h) { main_w = xev.xconfigure.width; main_h = xev.xconfigure.height; SubWindowsLayout(main_w, main_h, border_w, static_cast<int>(players.size()), layout_rects); for (auto i = 0; i < static_cast<int>(players.size()); ++i) { if (players[i]->GetWindow()) { XMoveResizeWindow(display, players[i]->GetWindow(), layout_rects[i].x_, layout_rects[i].y_, layout_rects[i].w_, layout_rects[i].h_); } } } } else { for (const auto& i: players) { assert(i); if (i->GetWindow() && i->GetWindow() == xev.xconfigure.window) { i->OnWindowSize(xev.xconfigure.width, xev.xconfigure.height); } } } } else if (xev.type == KeyPress) { if (xev.xkey.keycode == XKeysymToKeycode(display, XK_Escape)) { fprintf(stdout, "ESC Key Press\n"); for (const auto& i : players) { i->Stop(); if (i->GetWindow()) { XDestroyWindow(display, i->GetWindow()); i->SetWindow(None); } } players.clear(); XDestroyWindow(display, main_wid); XCloseDisplay(display); player_api.UnInit(); fprintf(stdout, "Close Players.... \n"); return 0; } } } } }Copy the code

Start play package

bool NT_PlayerSDKWrapper::Start(int buffer, bool is_mute, int render_scale_mode, bool is_only_dec_key_frame) { if (is_playing_) return false; if (url_.empty()) return false; if (! OpenHandle(url_, buffer)) return false; assert(handle_ && handle_->Handle()); Player_api_ ->SetMute(handle_->Handle(), is_mute? 1:0); player_api_->SetIsOutputAudioDevice(handle_->Handle(), 1); player_api_->SetAudioOutputLayer(handle_->Handle(), 0); Player_api_ ->SetVideoSizeCallBack(handle_->Handle(), this, &NT_Player_SDK_WRAPPER_OnVideoSizeHandle); player_api_->SetXDisplay(handle_->Handle(), display_); player_api_->SetXScreenNumber(handle_->Handle(),screen_); player_api_->SetRenderXWindow(handle_->Handle(), window_); player_api_->SetRenderScaleMode(handle_->Handle(), render_scale_mode); player_api_->SetRenderTextureScaleFilterMode(handle_->Handle(), 3); player_api_->SetOnlyDecodeVideoKeyFrame(handle_->Handle(), is_only_dec_key_frame ? 1:0); auto ret = player_api_->StartPlay(handle_->Handle()); if (NT_ERC_OK ! = ret) { ResetHandle(); return false; } is_playing_ = true; return true; }Copy the code

Stop playing

void NT_PlayerSDKWrapper::Stop() { if (! is_playing_) return; assert(handle_); player_api_->StopPlay(handle_->Handle()); video_width_ = 0; video_height_ = 0; ResetHandle(); is_playing_ = false; }Copy the code

Video width and height callback

extern "C" NT_VOID NT_CALLBACK NT_Player_SDK_WRAPPER_OnVideoSizeHandle(NT_HANDLE handle, NT_PVOID user_data,
	NT_INT32 width, NT_INT32 height)
{
	auto sdk_wrapper = reinterpret_cast<NT_PlayerSDKWrapper*>(user_data);
	if (nullptr == sdk_wrapper)
		return;

	sdk_wrapper->VideoSizeHandle(handle, width, height);
}
Copy the code

Real-time snapshot

extern "C" NT_VOID NT_CALLBACK NT_Player_SDK_WRAPPER_OnCaptureImageCallBack(NT_HANDLE handle, NT_PVOID user_data, NT_UINT32 result, NT_PCSTR file_name)
{
	auto sdk_wrapper = reinterpret_cast<NT_PlayerSDKWrapper*>(user_data);
	if (nullptr == sdk_wrapper)
		return;

	sdk_wrapper->CaptureImageHandle(handle, result, file_name);
}
Copy the code

Real-time mute

void NT_PlayerSDKWrapper::SetMute(bool is_mute)
{
	if (is_playing_ && handle_)
	{
		player_api_->SetMute(handle_->Handle(), is_mute?1:0);
	}
}
Copy the code

Setting drawing Mode

void NT_PlayerSDKWrapper::SetRenderScaleMode(int render_scale_mode) { if (is_playing_ && handle_) { player_api_->SetRenderScaleMode(handle_->Handle(), render_scale_mode); }}Copy the code

Set only keyframes to be solved

void NT_PlayerSDKWrapper::SetOnlyDecodeVideoKeyFrame(bool is_only_dec_key_frame) { if (is_playing_ && handle_) { player_api_->SetOnlyDecodeVideoKeyFrame(handle_->Handle(), is_only_dec_key_frame ? 1:0); }}Copy the code

The Handler management

bool NT_PlayerSDKWrapper::OpenHandle(const std::string& url, int buffer)
{
	if (handle_)
	{
		if (handle_->IsOpened()
			&& handle_->URL() == url)
		{
			return true;
		}
	}

	ResetHandle();

	auto handle = std::make_shared<NT_SDK_HandleWrapper>(player_api_);

	if (!handle->Open(url, buffer))
	{
		return false;
	}

	handle_ = handle;
	handle_->AddEventHandler(shared_from_this());

	return true;
}

void NT_PlayerSDKWrapper::ResetHandle()
{
	if (handle_)
	{
		handle_->RemoveHandler(this);
		handle_.reset();
	}
}
Copy the code

Other interfaces such as video recording are not repeated, but the Windows platform is consistent.

conclusion

Multi-channel RTMP or RTSP playback, involving performance and multi-channel audio and video synchronization, long-term playback stability and other issues, Linux platform can refer to less information, less options, interested in the reference.

​