Relative to rent, buy a house, decoration supervisor and other needs. In the past, we can only view the houses in different places through the plane pictures taken on the scene, but the details in this way are completely unable to feel, so we have to go to the ready-made house in person. So now more and more companies are starting to expand the field of panoramic viewing.

Panoramic picture consumption is very simple, as long as you have a mobile phone, a computer, you can browse panoramic pictures, panoramic video. On the content side, it’s a lot trickier. Fortunately, the current panoramic camera manufacturers all provide hardware and hardware integration interfaces, so that the third-party service companies can better expand the panoramic field of creativity and services. The panoramic camera in my hand is RICOH THETA SC2 for Business. The official interface document is also provided. The API annotation complies with the OSC by Google standard.

One, less wordy, see the effect first

~~The quality of the picture is relatively poor, it is good to see clearly~~

The mobile phone is a panoramic camera connected through wifi, which can capture the real-time preview of the camera and present a panoramic display in the mobile phone. When moving the camera, the picture will change in real time, and when dragging on the mobile phone, the picture can be displayed in different directions.

Camera API

1. Operation introduction

The camera is connected to the phone through the wifi function, and the camera is used as a wifi hotspot for the phone to connect. All the interfaces for requesting the camera can be directly requested to http://192.168.1.1

There are two types of Commands/Execute and Commands/Status. Both interfaces are of the POST type. As the interface name indicates, one is to perform the operation and the other is to view the Status of the operation. For example, to set the camera:

// Use the familiar Ajax example
$.ajax(
  type: 'POST'.url:'http://192.168.1.1/osc/commands/execute'.data: {
    "name": "camera.setOptions"."parameters": {
      "options": {
        "exposureProgram":1."iso":800."shutterSpeed":0.002}}})Copy the code

Name: The operation you want to perform
Parameters: Indicates the parameters of the operation

2, getLivePreview

This feature mainly uses the CAMERA. GetLivePreview interface of OSC. According to the official documentation, this API can only be used in the shooting mode of the SC model camera, and it will stop when the camera function is triggered or the shooting mode is changed

	parameter
Parameters	none
Output	Binary data of live view (MotionJPEG)

As you can see, this interface doesn’t need to fill in any parameters, it calls directly and returns a live View stream, which is a real-time MJPEG data stream. If you want to ask me what this very jPET-like thing is, let me just click on the table and add it later.

M-jpeg is a dynamic image compression technology developed based on the static image compression technology JPEG, which can generate a serialized moving image, each frame is a JPEG image

Iii. Implementation of Flutter

1. Packages to import

Two plug-ins are used to realize this function. The first one is used to make interface request and the second one is used to preview the panoramic image

HTTP: 0.12.2
Panorama: 0.3.1

You’re asking me why I don’t use Dio, which is a very powerful request tool. It’s because this interface returns a live Stream and requires a persistent connection method. HTTP provides a client for persistent connections, and the send method returns a StreamedResponse. But did not find in the other request library, hope to have to understand everyone to give directions.

In addition to the above two plug-ins, there are three official packages that are also required.

import 'dart:async'; // Asynchronous operation
import 'dart:typed_data'; // Use Uint8List
import 'dart:convert'; / / convert JSON
Copy the code

Declare 2 streams

Since you are requesting a stream of data, you need to declare a StreamSubscription to listen to the data for control. Then declare a StreamController to draw the data to the page

StreamSubscription vidoestream;
StreamController _streamController;
Copy the code

3. Make requests

Everything that needs to be introduced is introduced, and everything that needs to be declared is declared, so you can start requesting and processing data. As you can see, the request method is divided into two parts: the top half is used to make the request, and the bottom half is used to process the data into images

A, request

We need one firstclientInstance of can be calledsendMethod,
And this method needs to be based onBaseRequestInstance, and declare one morefinal req = http.Request();
RequestYou need to pass two more parameters(String method, Uri uri)
So in declaring aFinal URI = uri.http ('192.168.1.1', '/osc/commands/execute')
Finally usingclient.sendSend the request
Added atimeoutIs to do timeout processing

void liveStream() async {
    // 1
    final client = http.Client();
    final uri = Uri.http('192.168.1.1'.'/osc/commands/execute');
    final params = {"name": "camera.getLivePreview"};
    final req = http.Request('post', uri);
    req.body = json.encode(params);
    final res = await client.send(req).timeout(Duration(seconds: 5));
    
    
    // 2. Data conversion
    const _trigger = 0xFF;
    const _soi = 0xD8;
    const _eoi = 0xD9;
    //1, declare an empty List of integers
    List<int> chunks = <int> [];//2. Subscribe to the data stream returned by the request
    vidoestream = res.stream.listen((List<int> data) async {
      if(chunks.isEmpty) { // Check whether the current chunks have data
        final startIndex = data.indexOf(_trigger); // Insert the first chunk into the chunks
        if(startIndex >=0 && startIndex+1 < data.length && data[startIndex +1] == _soi) {
          final slicedData = data.sublist(startIndex, data.length);
          / / 3, insertchunks.addAll(slicedData); }}else {
        final startIndex = data.lastIndexOf(_trigger); // Insert the last chunk, indicating that the frame is complete
        if( startIndex + 1 < data.length && data[startIndex + 1] == _eoi ) {
          final slicedData = data.sublist(0, startIndex + 2);
          / / 3, insert
          chunks.addAll(slicedData);
          //4 Convert it to an image and add it to stream
          final imageMemory = MemoryImage(Uint8List.fromList(chunks));
          await precacheImage(imageMemory, context);
          _streamController.add(imageMemory);
          //5 Clear the data of this frame
          chunks = <int> []; }else { // Neither the beginning nor the end, the middle chunk is inserted directly
          / / 3, insertchunks.addAll(data); }}}); }Copy the code

Two, conversion data

For those of you who don’t know MJPEG, this is probably the most confusing place.

There are three points in point 3 in the note above, so let’s take a look at the five points

1, List chunks = [];
2. Res. Stream. Listen
3, chunks. AddAll
4, _streamController. Add (imageMemory)
5, chunks = []

List

to store the image encoded data. In the LISTEN res.stream, process the data of each frame and insert it into the chunk using addall method. We convert the data to imageMemory, insert it into _streamController, and preview in real time while the data is constantly updated.

Insert page

Here is relatively simple, direct use of a StreamBuilder control, which is wrapped in the use of Panorama, the function is implemented.

StreamBuilder(
  stream: _streamController.stream,
  builder: (context, db) {
    if(db.hasData) {
      return Panorama(
        child: Image(image: db.data),
      );
    }
    return Text('No data'); },),Copy the code

All right, let's finish sprinkling.Copy the code

Wait, it’s not that easy. There are two very important questions:

_soi,_eoiWhat is it?
whychunks.addAllThree times?

Let’s move on

Four, the core principle understanding

In order to solve the problem, only spent these two days to see the relevant materials, with you to understand, and dare not say to explain 😂

1, MJPEG

Looking at the official explanation, we can know that the data we go back to each frame is the data of a JPEG, so we only need to convert the data to the picture, so the question is, how do we convert the data to the picture?

Motion JPEG(M-JPEG or MJPEG, Motion Joint Photographic Experts Group, FourCC:MJPG) is an image compression format in which each frame is individually encoded using JPEG. M-jpeg is commonly used in image acquisition devices such as digital cameras and webcams. MJPEG is dynamic JPEG, which uses JPEG compression algorithm to compress video signals at a speed of at least 25 frames per second.

If we look at the official interpretation of JPEG, we also see that JFIF(JPEG File Interchange Format) is used as the standard to determine which markup codes are in the data and to process the data

JPEG committee in the formulation of JPEG standard, the definition of many marker code (marker) or (marker segments) composition, used to distinguish and identify image data and related information. Currently, the most widely used Interchange Format is JFIF(Jpeg File Interchange Format). Each tag code in JPEG consists of two bytes, preceded by a fixed value of 0xFF, and each tag code can be preceded by an unlimited number of 0xFF padding bytes. The bytes in a JPEG file are arranged in positive order, that is, the higher-order bytes are first and the lower-order bytes are second.

And if we look at what the data looks like,

JFIF:

Tag code	The numerical	describe
SOI(start of image)	FFD8	Image began to
EOI(end of image)	FFD9	End of the image

At present, we only need to encode the location of the image development and end. We can find the index of the first byte in FF D8 and the index of the last byte in FF D9, and then intercept the data in the middle, which is the data we need for the graph

List<int> chunks = <int> [];const _trigger = 0xFF;
const _soi = 0xD8;
const _eoi = 0xD9;
int startIndex = - 1;
int endIndex = - 1;
// Data is the data stream we requested
if(data[i] == _trigger && data[startIndex + 1] == _soi ) {
  startIndex = i
}
if(data[i] == _trigger && data[startIndex + 1] == _eoi ) {
  endIndex = i
}
chunks = data.sublist(startIndex, endIndex)
Copy the code

Okay, so we've done this again. No husband.Copy the code

I plug this data into a MemoryImage, run the program, and display it on the page, only to find that each time the Image is incomplete and the console sends an Invalid Image error each time.

Print (startIndex) or print(endIndex) will find that the initial value -1 is printed multiple times, and once the correct index position appears, print -1 multiple times, and so on. This indicates that the data of each frame is not complete, and is divided into multiple blocks. It is necessary to combine these data blocks to obtain a complete image of a frame.

2, the Transfer – Encoding: chunked

Looking at the general MJPEG-streaming implementation and the HTTP. Dart source code in the Flutter plugin market, we found that there was not much data processing on the server or client side. The problem was that we printed the HTTP response message during transmission. You can see that the response header looks like this.

key	value
Connection	close
X-Content-Type-Options	nosniff
Content-Type	multipart/x-mixed-replace; boundary=”—osclivepreview—“
Transfer-Encoding	chunked

Transfer-encoding: chunked means that when a large amount of data is transmitted, the data is split into multiple chunks so that the page is displayed step by step. This ability to block the entity body is called block transfer coding.

So you can understand why when you print the transmitted data, you print SOI and EOI every few times, so you just need to stitch the data together. Go back to the previous data processing method, and take a look at this method again. It is completely understood.

const _trigger = 0xFF; / / logo
const _soi = 0xD8; // Start the image
const _eoi = 0xD9; // The image ends
List<int> chunks = <int> [];// to save the data of each frame
vidoestream = res.stream.listen((List<int> data) async {
  // Check whether the current chunks have data
  if(chunks.isEmpty) { 
    // Find the starting identifier
    final startIndex = data.indexOf(_trigger); 
    if(startIndex >=0 && startIndex+1 < data.length && data[startIndex +1] == _soi) {
      // From the beginning of the identification, to the end of the useful data
      finalslicedData = data.sublist(startIndex, data.length); chunks.addAll(slicedData); }}else {
    // Find the end identifier,
    final startIndex = data.lastIndexOf(_trigger);
    if( startIndex + 1 < data.length && data[startIndex + 1] == _eoi ) {
      // From the beginning to the end identifier is useful data
      final slicedData = data.sublist(0, startIndex + 2);
      // Insert, and the data for that frame is complete
      chunks.addAll(slicedData);
      final imageMemory = MemoryImage(Uint8List.fromList(chunks));
      await precacheImage(imageMemory, context);
      _streamController.add(imageMemory);
      chunks = <int> []; }else { 
      // Insert data that is not at the beginning, end or in the middlechunks.addAll(data); }}});Copy the code

In daily work, I didn’t use it much and completely ignored it. In order to find where the data was divided into blocks, I spent a lot of time on research. Does the camera block the data, or does THE HTTP dart help me block it, and it turns out I’ve learned a lot about the HTTP protocol.

5, summary

The implementation of this function takes the longest time. This part involves a lot of knowledge, such as the use of StreamController in Flutter, how to request a MJPEG stream, image coding technology, and HTTP block coding transmission. However, the basic knowledge is not solid, so we need to find the information to learn. Fortunately, after this development, also learned a lot of things, to understand the importance of basic knowledge. The realization of this function, in preparation for this article, there may be a lot of deficiencies, welcome to corrections.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

【MJPEG】Flutte develops panoramic cameras

One, less wordy, see the effect first

Camera API

1. Operation introduction

2, getLivePreview

Iii. Implementation of Flutter

1. Packages to import

Declare 2 streams

3. Make requests

A, request

Two, conversion data

Insert page

Four, the core principle understanding

1, MJPEG

2, the Transfer – Encoding: chunked

5, summary

【MJPEG】Flutte develops panoramic cameras

One, less wordy, see the effect first

Camera API

1. Operation introduction

2, getLivePreview

Iii. Implementation of Flutter

1. Packages to import

Declare 2 streams

3. Make requests

A, request

Two, conversion data

Insert page

Four, the core principle understanding

1, MJPEG

2, the Transfer – Encoding: chunked

5, summary

Related Posts

Vue with Echarts for simple data visualization development 02

Front-end cornerstones: pre-processing mechanism, variable enhancement

Math.random() generates random data in JavaScript