background
In my work, I found that many live CDN implementations of HTTPFLV pull do not use HTTP chunk encoding, but directly use the no-content-length approach. So I want to build a live CDN that supports HTTP chunk encoding.
Environment set up
Ubuntu 18.04.4 LTS software nginx-1.18.0 Nginx extension module nginx-http-flv-module
Nginx – HTTP – FLV – modulex download
git clone https://github.com/winshining/nginx-http-flv-module.git
Copy the codeNginx configuration installation
./configure --add-module=/home/wanghao/worker/opensourcecode/nginx_module/nginx-http-flv-module
make -j 4 && make install
cd /usr/local/nginx/
Copy the codeNginx. conf configuration file
#user nobody;
worker_processes 1;
events {
worker_connections 1024;
}
http { ## HTTP pull configuration
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name www.wawa.com;
location /live {
flv_live on;
}
}
}
rtmp { ## RTMP push stream configurationserver { listen 1935; application myapp{ live on; record off; }}}Copy the codeIn cases where the length of the file is not known (such as live streaming), nginx uses chunked_transfer_encoding by default. Therefore, you do not need to set what is displayed in the configuration file
Syntax: chunked_transfer_encoding on | off; Default: chunked_transfer_encoding on; Context: http, server, locationCopy the codePlease click the official link
Ffmepg pushes the local file to the live broadcast server
Ffmpeg - re - I q00307z84wz. 321002.1 ts-fFLV RTMP: / / 192.168.116.130:1935 / myapp / 123Copy the codeQ00307z84wz.321002.1. ts is a 5-minute file, so ffmpeg exits after about 5 minutes.
Real 5M0.382s user 0M22.750s SYS 0M8.141sCopy the codeHTTPFLV pull flow
Enter the correct IP address, port, and stream ID to pull the stream
# curl - L - v "http://192.168.116.130/live? port=1935&app=myapp&stream=123" -o 1.flv
* Trying 192.168.116.130...
* TCP_NODELAY set% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 - : -- : -- - : -- : -- -- - : - 0 * Connected to 192.168.116.130 (192.168.116.130) port 80 (# 0)> GET /live? Port =1935&app=myapp&stream=123 HTTP/1.1 > Host: 192.168.116.130 > user-agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.18.0 < Date: Fri, 03 Jul 2020 13:49:52 GMT < content-type: video/x-flv < Transfer-Encoding: chunked < Connection: keep-alive < Expires: -1 < { [449 bytes data] 100 1068k 0 1068k 0 0 97k 0 --:--:-- 0:00:10 --:--:-- 117kCopy the codeYou can see in the response header that the HTTP chunk protocol is used and there is no content-length
< Transfer-Encoding: chunked
Copy the codeContinue to modify nginx.conf to close the HTTP chunk encoding
location /live {
flv_live on;
chunked_transfer_encoding off;
}
Copy the codeUse curl to pull the stream test
# curl -L -v "Http://192.168.116.130/live? port=1935&app=myapp&stream=123" -o 1.flv
* Trying 192.168.116.130...
* TCP_NODELAY set
%Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.116.130 (192.168.116.130) port 80 (# 0)>GET /live? Port = 1935 & app = myapp&stream = 123 HTTP / 1.1
> Host: 192.168.116.130
>The user-agent: curl / 7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.18.0
< Date: Fri, 03 Jul 2020 13:59:43 GMT
< Content-Type: video/x-flv
< Connection: keep-alive
< Expires: -1
* no chunk, no close, no size. Assume close to signal end
<
{ [1504 bytes data]
100 668k 0 668k 0 0 98k 0 --:--:-- 0:00:06 --:--:-- 112k
Copy the codeYou can see that the “transfer-encoding: chunked” response header is no longer there. If you look closely, it prints a line in English
* no chunk, no close, no size. Assume close to signal end
Copy the codeNo size = content-length = no chunk = transfer-encoding: If you Assume close to signal end, the live stream will end. If you Assume close to signal end, the live stream will end. If you Assume close to signal end, the live stream will end. So the no-centent-length scheme is non-standard.
“Connection: keep-alive”. In alive broadcast, the Connection is usually close. HTTP1.1 uses keep-alive by default, but clients can specify the use of HTTP short links in the header of the request.
root@PF1YTXHH-ZTB:/storage/log_live# curl - L - H "Connection: Close" - v "http://192.168.116.130/live? port=1935&app=myapp&stream=123" -o 1.flv
* Trying 192.168.116.130...
* TCP_NODELAY set% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 - : -- : -- - : -- : -- -- - : - 0 * Connected to 192.168.116.130 (192.168.116.130) port 80 (# 0)> GET /live? Port =1935&app=myapp&stream=123 HTTP/1.1 > Host: 192.168.116.130 > user-agent: curl/7.58.0 > Accept: */* > Connection:Close# Client specify use close mode< HTTP/1.1 200 OK < Server: nginx/1.18.0 < Date: Tue, 07 Jul 2020 13:14:43 GMT < content-type: video/x-flv < Connection: closeThe server also returns close
< Expires: -1
<
{ [1637 bytes data]
100 236k 0 236k 0 0 89583 0 --:--:-- 0:00:02 --:--:-- 89583^C
Copy the codeTest other live streaming apps
Using ADB Logcat to capture the logs of apps such as Fast Hand, Douyu, Huya, etc., and find the pull address. Using curl to pull the stream, I also found that many streams are no-content-length. The reason for using this non-standard no-Centengt-length is implementation simplicity.
Unfinished business
When an HTTP chunk stream is pulled, the PACKETS captured by using tcpdump and analyzed by Wireshark are not detected by the HTTP chunk protocol, which is inconsistent with other online tutorials. It needs a little more exploration.
Tcpdump -i lo host 192.168.116.130 -nnvV -w live2.pcapCopy the codeBecause the nginx server is on the same machine as the stream pulling terminal, the captured packets are pulled from the local loopback interface (LO). 192.168.116.130 is the local Intranet IP address.