Make writing a habit together! This is my first day to participate in the “Gold Digging Day New Plan · April More text challenge”, click to see the details of the activity.

Through this article, you will learn: 1. Knowledge and techniques related to Flutter 2. Techniques related to Flutter operation 3. Commercial scenarios, the mainstream implementation of audio and video requirements

Writing background

A recent project is a fitness-based Flutter App whose core function is the playback of training courses. Due to the poor user experience caused by various factors, I reformed this function twice during the six months I took over the project, and pre-developed a set of long-term plans to support the continuous iteration of the core function of course playback. So record to share with everyone, to avoid repeated trample pit detour. To be clear: the hard part of audio and video (or indeed any technical problem) is the solution, not any specific implementation code; The solution is often not achieved overnight, because it involves a lot of problems such as front-end requirements, development resources, team resources and so on. Next, I will focus on the whole exploration process of realizing the solution of audio and video requirements.

The bumpy road to implementation

It is never too late to mend

In the early stage of the project, the courses of App were directly played online without caching mechanism. During user training, there was a situation that training lasted for 5 seconds and waited for 10 seconds, and each time it was loaded online, which greatly consumed traffic. Therefore, we analyzed the video and found that the bit rate reached 16000KBps +, so that the video of just 30 seconds has 60MB, which is the reason why the buffering is always in the process of playing. Meanwhile, the index information of the video is not optimized, moov Atom is placed at the end, and the decoding speed of the player is slow, resulting in a long waiting time for the first time. Due to operational capacity, you can only use compression tools to reduce the video size first. I urgently launched a download function to cache all the videos of course chapters locally before training, and play them using local resources during training. The technical aspect of Flutter is to download sequentially using Dio Download directly.

Ii. Keep pace with The Times

Through the temporary online download function, users can smoothly train; However, it soon leads to the next problem: the video size is not enough, 10 chapters may need 5 min+ download time, at the same time, the integrity of the file is not verified, the error rate is high. How to solve it?

  1. Start at the source: Through a lot of communication with editors and continuous demo production to check the effect, we found a size ofMp4 is 1080 * 1080. The bit rate is around 2000 and the peak frame rate is 25On any phone screen with any resolution/DPI, the smoothness and sharpness are perfectly fine. Then through professional compression tools, basicVideo per minute can be controlled under 8M. So ask your content team to make a video of this size.
  2. Optimize download mechanismThe download mechanism was changed from simple and crude to sequential download:Download the first chapter, enter the training page will start the background download, at the same time support breakpoint download.

    This reduces the user’s waiting time for downloading, and avoids the invalidity of the original unfinished downloaded chapter when the user cuts the chapter, saving the traffic of repeated downloading. (Mainly also to the server throttling 😊)

    Here is the implementation of Flutter breakpoint download:

    – Files downloaded during the process are terminated with the suffix.mp4.temp. Files downloaded after the process are terminated with mp4.

    – Read the length of the incomplete file download for this resource in the local cache;

    – Set the download length to "range": "bytes=$downloadStart-";

    – Notifies the caller of the download progress via stream.
Future<void> downloadFile({
  required String url,
  required String savePath, // Local cache path
  required CancelToken cancelToken, // Download credentials are passed in by the caller to operate the download node (e.g., cancel)
  ProgressCallback? onReceiveProgress,
  void Function()? done,
  void Function(Exception)? failed,
}) async {
  int downloadStart = 0;
  File f = File(savePath);
  if (await f.exists()) {
    downloadStart = f.lengthSync();
  }
  print("start: $downloadStart");
  try {
    var response = await downloadDio.get<ResponseBody>(
      url,
      options: Options(
        /// Receive response data as a stream
        responseType: ResponseType.stream,
        followRedirects: false,
        headers: {
          /// Downloading key locations in segments
          "range": "bytes=$downloadStart-",})); File file = File(savePath); RandomAccessFile raf = file.openSync(mode: FileMode.append);int received = downloadStart;
    int total = await_getContentLength(response); Stream<Uint8List> stream = response.data! .stream; StreamSubscription<Uint8List>? subscription; subscription = stream.listen( (data) {/// Write files must be synchronizedraf.writeFromSync(data); received += data.length; onReceiveProgress? .call(received, total); }, onDone: ()async {
        file.rename(savePath.replaceAll('.temp'.' '));
        awaitraf.close(); done? .call(); }, onError: (e)async {
        awaitraf.close(); failed? .call(e); }, cancelOnError:true,); cancelToken.whenCancel.then((_)async {
      awaitsubscription? .cancel();await raf.close();
    });
  } on DioError catch (error) {
    if (CancelToken.isCancel(error)) {
      print("Download cancelled");
    } else{ failed? .call(error); }}}Copy the code
  1. File integrity verification: Md5 encoding is used to verify file integrity. Md5 encoding is used when the operating platform uploads the video, and MD5 encoding is used when the App downloads the video successfully. If the two are the same, the file is judged to be complete.
File(path).readAsBytes().then((Uint8List str) {
  if (md5.convert(str).toString() == md5Str) {
    // md5Str is the code returned by the server
    // The file is complete}});Copy the code

Looking to the future

After two times of optimization, it can basically meet the use of current users. But it is far from enough. I want to Keep the bid to meet the following expectations:The video can be turned on in seconds, play smoothly, save traffic, improve security (for example, have their own encoding format), and differentiate bit streams (choose the best solution under different sizes/networks).

In general, a video stream needs to go through the process from loading to preparing for playbackUnlock protocol, unlock package and unlock codeSuch a process, in which the agreement is made,We're using HTTP; Packaging is the format, such as MP4, M3U8, RMVB, etc. The format is usually the package of a set of encoding format, such as MP4 encoding format is mostly H264, H265; M3u8 is HLS; RMVB is mainly RV40;

  • Make sure the container

    I hope to achieve the effect of code stream differentiation. When the client plays the top-level M3U8 file, it will choose the stream with high bit rate, and when the bit rate cannot reach the stream, it will request the stream with low bit rate. Therefore, M3U8 itself can be directly used for multi-bit rate video. soDetermine to use m3U8 format for video storage.

    How to convert m3U8 format?

    1. When uploading videos, the operating platform uses FFMPEG to encode source files and ADAPTS multi-bit streams;

    2. The server side selects transcoding and dynamic bit rate support. Qiuniuyun and Ali Cloud have similar support to unify all encoding and compression through transcoding and provide multi-bit rate video.
  • 1. The caching mechanism is a difficulty. M3u8 is a collection of TS files. 2. When the integrity check of TS files fails, it needs to request online playback. This fault-tolerant mechanism is also a problem. 3. The technology of downloading while broadcasting, this I have not yet to understand, but it is certainly a difficult 😄; .
  • Improved security

    In fact, iQiyi and Tencent Video have their own encoding formats, but our project is not a complete video playing application at present, so I think it is enough to encrypt and decrypt slices, for exampleAES-128.
  • Open video seconds

    The optimization of video seconds on mainly lies inNumber of successfully loaded plays/total plays in one second, we know that M3U8 is composed of multiple slices, so the second opening is closely related to the size of the first slice, generallyThe length of the first TS clip is controlled at 1~3 seconds.

    The first frame of the video (similar to the concept of keyframe) should make up the picture. It is best to wrap the key frame when reducing the bit rate in the process of video editing (I also heard it through hearsay, and only after doing video editing operations can I understand 😂).

Write in the last

In fact, the current mainstream scheme mostly contains these technical points: M3U8 format, HLS coding, AES-128 encryption and decryption, while playing, key frame control……

Currently, the mainstream libraries for Flutter are GSYVideoPlayer on Android and XgPlayer on Web. Unfortunately, there is no good library for Flutter. Currently, we use the official videoPlayer, and we are looking forward to more open source libraries.

As for the exploration of audio and video, I am also in the process of learning step by step, looking forward to more discussions on this article!

Thank you very much

Big GSY answers the questions himself

Bytecode and the Junichiro helped out

Explore mobile audio and video and GSYVideoPlayer tour

Ffmpeg multi-bit rate M3U8 format conversion