In the live broadcast system, in addition to live audio and video, sometimes text information is released from the host end, which can be sent to the user’s player end without the video transmission channel. However, if the transmitted data wants to keep accurate synchronization with the video, The best way is to package the information and video data together and transfer the data into the H264 Access Unit through the H264 SEI mode.
H264 SEI
SEI stands for Supplemental Enhancement Information
SEI Nal Unit Type: 6
SEI syntax definitions:
Supplemental Enhancement Information RBSP syntax
| sei_rbsp( ) { | C | Descriptor |
|---|---|---|
| do | ||
| sei_message( ) | 5 | |
| while( more_rbsp_data( ) ) | ||
| rbsp_trailing_bits( ) | 5 | |
| } |
Supplemental Enhancement Information Message Syntax:
| sei_message( ) { | C | Descriptor |
|---|---|---|
| payloadType = 0 | ||
| while( next_bits( 8 ) = = 0xFF ) { | ||
| ff_byte /* equal to 0xFF */ | 5 | f(8) |
| payloadType += 255 | ||
| } | ||
| last_payload_type_byte | 5 | u(8) |
| payloadType += last_payload_type_byte | ||
| payloadSize = 0 | ||
| while( next_bits( 8 ) = = 0xFF ) { | ||
| ff_byte /* equal to 0xFF */ | 5 | f(8) |
| payloadSize += 255 | ||
| } | ||
| last_payload_size_byte | 5 | u(8) |
| payloadSize += last_payload_size_byte | ||
| sei_payload( payloadType, payloadSize ) | 5 | |
| } |
SEI semantics:
Supplemental enhancement information RBSP semantics
Supplemental Enhancement Information (SEI) contains information that is not necessary to decode the samples of coded pictures from VCL NAL units.
Supplemental enhancement information message semantics
An SEI NAL unit contains one or more SEI messages. Each SEI message consists of the variables specifying the type payloadType and size payloadSize of the SEI payload. SEI payloads are specified in Annex D. The derived SEI payload size payloadSize is specified in bytes and shall be equal to the number of bytes in the SEI payload.ff_byte is a byte equal to 0xFF identifying a need for a longer representation of the syntax structure that it is used within. last_payload_type_byte is the last byte of the payload type of an SEI message.
last_payload_size_byte is the last byte of the size of an SEI message
An Sei Nal Unit can contain multiple Sei messages. Each Sei message has a payloadType. When payloadType is set to 5 in H264, sei_playload allows users to customize data. Then we can use it to transmit data.
This is the end of the SEI basics, but if you want to implement your own code, you will need to learn more details. I recommend that you read the H264 documentation carefully. Then come to practice.
Download the software first: github.com/daniulive/S… For testing purposes, download the Windows version.
RTMP transmits text messages:
1. Start the push software smartPublisherDemo. exe
2. Perform the following configuration:
3. You can click the auto send text button
4. Open SmartPlayer.exe on the player end to view the effect of data transmission.
RTSP transmits text messages:
1. Start the push software smartPublisherDemo. exe
2. Perform the following configuration:
3. You can click the auto send text button
4. Open SmartPlayer.exe on the player end to view the effect of data transmission.
conclusion
The first advantage is that it does not depend on related protocols. RTSP and RTMP can be used, and other protocols can be used as long as the player supports SEI parsing. The second is good compatibility. If the player does not support custom SEI data parsing, the SEI data is thrown to the H264 decoder, which simply ignores it without affecting normal playback. The above operation can also be played with VLC, and it works fine, except that the SEI message is not displayed. The third is to be completely in sync with the video, something no other channel can do.