How to build video transcoding cluster and play server

This article is published by authorized NetEase Cloud community by author Zhang Yuandao.

Building transcoding cluster

With the development of media technology, the popularity of hardware devices and the extension of mobile devices in every corner of society, people can generate information anytime and anywhere. And in these massive information, have a lot of multimedia information. Multimedia information has become an important way for people to share information. It is no exaggeration to say that this is an age of information explosion. Traditional TV, film and video websites, as well as mobile “we media” (free people) are constantly spitting out media information.

The existence of this massive amount of information brings challenges to storage and sharing. The challenge is twofold. One is how to store this information as cheaply as possible. The other is how to make the information generated by these disparate devices viewable by different devices. The solution to problem one is to minimize the amount of multimedia information without reducing the quality of media information. The solution to the second problem is to encode the videos of different formats produced by different devices into a unified video format that can support different devices.

Fortunately, there is a solution that has been proven to work in practice: video coding. A video coding cluster can be constructed to encode the original video into the target format. In the process of re-coding, video compression and unified format can be completed, so as to achieve storage reduction and unified format. From the perspective of video, current devices can be divided into two camps, one is Apple camp, the other is non-Apple camp. Apple camp video as long as m3U8 as the package format, internal code with H264 and AAC can be. And the non-Apple camp is MP4 for the package format, internal coding has H264 and AAC can be. According to preliminary statistics, H264+ AAC encoding combination can achieve 2:1 to 4:1 compression ratio, which can significantly reduce the storage space.

Below is a simple transcoding cluster built using the Gearman framework (of course, this simple cluster took one or two people a month to build and deploy in production).

The simple transcoding cluster consists of four modules:

The JobServer receives requests from the API server, creates transcode jobs, and sends jobs to transcode-worker.
The API server receives transcoding requests from third-party products and forwards them to JobServer.
Transcode-worker receives the function request from JobServer and completes the transcoding job.
A distributed file system is used to store original and target videos. The distributed file system is used because Transcode-worker is deployed on different servers, and Transcode-worker needs to be able to access video files in a distributed environment.

Required technology:

Gearman distributed task scheduling framework. Very lightweight, supports C, Java, Python, PHP, C#, Ruby, etc. People with any language background can work together.
Ffmpeg, the Swiss Army knife of transcoding. Supports almost every commercial video format on the market. Except for laboratory formats.
Build API web server technology. I am more used to J2EE Spring that set. Of course you can use Ruby on Rails,.net, etc.
Basic knowledge of audio and video. Including H264 coding technology, video decompression knowledge.

At this point, a content production system is basically in place. But you also need a deliver system for video content, a video playback server [cluster]. The steps of setting up the video server will be recorded in detail here.

How to set up a FLV media server

The environment

Operating system: Linux 2.6.32-5-AMd64. Linux version is ok.

Server: nginx 1.3.3. Nginx versions can be higher.

Client: jwPlayer

Set up steps

Install nginx with http_module_flv

1. Download nginx

Wget HTTP: / / http://nginx.org/download/nginx-1.3.3.tar.gz

2, decompression

The tar – XZVF nginx – 1.3.3. Tar. Gz

3, installation,

CD nginx – 1.3.3

./configure –prefix=/home/user/nginx_flv –with-http_flv_module

4, configuration,

#user nobody;

worker_processes 1;

#error_log logs/error.log;

#error_log logs/error.log notice;

#error_log logs/error.log info;

#pid logs/nginx.pid;

events {

worker_connections 1024;

}

http {

include mime.types;

default_type application/octet-stream;

Log_format main ‘remoteADDR −remote_user [timelocal]”request”

‘statusbody_bytes_sent “$http_referer” ‘

‘”httpuseragent””http_x_forwarded_for”‘;

#access_log logs/access.log main;

sendfile on;

#tcp_nopush on;

#keepalive_timeout 0;

keepalive_timeout 65;

#gzip on;

server {

listen 8088;

server_name localhost;

#charset koi8-r;

access_log logs/host.access.log main;

location / {

root /home/fs/video;

index index.html index.htm;

autoindex on;

location ~ \.flv{

flv;

}

#error_page 404 /404.html;

# redirect server error pages to the static page /50x.html

error_page 500 502 503 504 /50x.html;

location = /50x.html {

root html;

}

# deny access to .htaccess files, if Apache’s document root

# concurs with nginx’s one

#location ~ /\.ht {

# deny all;

}

Note: The green part is configured with FLV. The /home/fs/video directory is the directory where resources, including jwPlayer and test videos, are stored.

5. Configure jwPlayer server.

Download the open source JwPlayer, such as MediaPlayer-5.10. Unzip player. SWF and jwplayer.js to /home/fs/video. And put the test video zyd. FLV in /home/fs/video as well.

6, test,

Type http://hostip:8080/player.swf? in your browser type=http&file=zyd.flv

You can drag the progress bar to view the video playing normally.

How to buildMP4 Media Server

The environment

Operating system: Linux 2.6.32-5-AMd64. Linux version is ok.

Server: nginx 1.3.3. Nginx versions can be higher.

Client: jwPlayer. Or any player.

Set up steps

Install nginx with nginx_mod_h264_streaming

1, Download nginx and nginx_mod_h264_Streaming

Wget nginx.org/download/ng…

Git clone github.com/vivus-ignis…