How to upload a large file to a breakpoint?

What is it

Requirements, no matter how simple, become extremely complex at a certain level of magnitude. Right

Uploading files is easy, but getting bigger is complicated

There are several variables that affect our user experience when uploading large files

  • The ability of a server to process data
  • The request timeout
  • Network fluctuation

Upload time will be longer, frequent file upload failure, failure and need to upload again, etc

To solve the above problem, we need to handle large file uploads separately

There are two concepts of fragment upload and resumable

Shard to upload

Fragment upload is to divide the file to be uploaded into multiple data blocks (parts) based on a certain size

The following figure

After uploading, the server collects and integrates all uploaded files into original files

The general process is as follows:

  1. Divide the files to be uploaded into data blocks of the same size according to certain segmentation rules.
  2. Initialize a fragment upload task and return the unique identifier of the fragment upload.
  3. Send each fragmented data block according to a certain strategy (serial or parallel);
  4. After the data is uploaded, the server determines whether the data is uploaded completely. If yes, the server synthesizes data blocks to obtain the original file

Breakpoint continuingly

Broken upload refers to the artificial division of the download or upload task into several parts when downloading or uploading

Each part uses a thread to upload or download. In case of network failure, you can continue to upload and download the unfinished part from the uploaded or downloaded part, instead of starting from the beginning. Users can save time and speed up

There are two general implementations:

  • The server returns telling you where to start
  • The browser handles this by itself

During the upload process, the file is written as a temporary file on the server. After all the files are written (uploaded), the temporary file is renamed as an official file

If the upload is interrupted, the next upload will be based on the current temporary file size as the offset of the file read on the client, continue to read file data blocks from this location, and upload to the server continue to write files from this offset

Second, implementation ideas

The overall idea is relatively simple: get the file, save the unique identifier of the file, cut the file, upload the file in sections, upload a section at a time, judge the uploading progress according to the unique identifier until all the fragments of the file are uploaded

Everything below is pseudo-code

Read file contents:


const input = document.querySelector('input');
input.addEventListener('change'.function() {
    var file = this.files[0];
});
Copy the code

You can use MD5 to achieve file uniqueness

const md5code = md5(file);
Copy the code

Then start to split the file

var reader = new FileReader();
reader.readAsArrayBuffer(file);
reader.addEventListener("load".function(e) {
    // Cut a section every 10M, here is only one cutting demonstration, the actual cutting needs to cycle cutting,
    var slice = e.target.result.slice(0.10*1024*1024);
});
Copy the code

H5 Upload one (piece)

const formdata = new FormData();
formdata.append('0', slice);
// There is a pit where some devices cannot get the file name and file type. This is addressed at the end
formdata.append('filename', file.filename);
var xhr = new XMLHttpRequest();
xhr.addEventListener('load'.function() {
    //xhr.responseText
});
xhr.open('POST'.' ');
xhr.send(formdata);
xhr.addEventListener('progress', updateProgress);
xhr.upload.addEventListener('progress', updateProgress);

function updateProgress(event) {
    if (event.lengthComputable) {
        / / the progress bar}}Copy the code

Here is a common picture and video file type judgment

function checkFileType(type, file, back) {
/**
* type png jpg mp4 ...
* file input.change=> this.files[0]
* back callback(boolean)
*/
    var args = arguments;
    if(args.length ! =3) {
        back(0);
    }
    var type = args[0]; // type = '(png|jpg)' , 'png'
    var file = args[1];
    var back = typeof args[2] = ='function' ? args[2] : function() {};
    if (file.type == ' ') {
        // If the system cannot get the file type, it reads the binary stream to parse the file type
        var imgType = [
            'ff d8 ff'.//jpg
            '89 50 4e'.//png

            '0 0 0 14 66 74 79 70 69 73 6F 6D'.//mp4
            '0 0 0 18 66 74 79 70 33 67 70 35'.//mp4
            0 0 0 0 66 74 79 70 33 67 70 35'.//mp4
            '0 0 0 0 66 74 79 70 4D 53 4E 56'.//mp4
            '0 0 0 0 66 74 79 70 69 73 6F 6D'.//mp4

            '0 0 0 18 66 74 79 70 6D 70 34 32'.//m4v
            '0 0 0 0 66 74 79 70 6D 70 34 32'.//m4v

            '0 0 0 14 66 79 70 71 74 20 20'.//mov
            '0 0 0 0 66 74 79 70 71 74 20.//mov
            '0 0 0 0 6D 6F 6F 76'.//mov

            '4F 67 67 53 0 02'.//ogg
            '1A 45 DF A3'.//ogg

            '52 49 46 46 x x x x 41 56 49 20'.//avi (RIFF fileSize fileType LIST)(52 49 46 46,DC 6C 57 09,41 56 49 20,4C 49 53 54)
        ];
        var typeName = [
            'jpg'.'png'.'mp4'.'mp4'.'mp4'.'mp4'.'mp4'.'m4v'.'m4v'.'mov'.'mov'.'mov'.'ogg'.'ogg'.'avi',];var sliceSize = /png|jpg|jpeg/.test(type) ? 3 : 12;
        var reader = new FileReader();
        reader.readAsArrayBuffer(file);
        reader.addEventListener("load".function(e) {
            var slice = e.target.result.slice(0, sliceSize);
            reader = null;
            if (slice && slice.byteLength == sliceSize) {
                var view = new Uint8Array(slice);
                var arr = [];
                view.forEach(function(v) {
                    arr.push(v.toString(16));
                });
                view = null;
                var idx = arr.join(' ').indexOf(imgType);
                if (idx > -1) {
                    back(typeName[idx]);
                } else {
                    arr = arr.map(function(v) {
                        if (i > 3 && i < 8) {
                            return 'x';
                        }
                        return v;
                    });
                    var idx = arr.join(' ').indexOf(imgType);
                    if (idx > -1) {
                        back(typeName[idx]);
                    } else {
                        back(false); }}}else {
                back(false); }}); }else {
        var type = file.name.match(/.(\w+)$/) [1]; back(type); }}Copy the code

Call as follows

checkFileType('(mov|mp4|avi)',file,function(fileType){
    // fileType = mp4,
    // Return false if the type of file is not included in the enumeration
});
Copy the code

The upload step above can be changed to:

formdata.append('filename', md5code+'. '+fileType);
Copy the code

With the cut upload, there is also a unique file identification information, breakpoint continuation becomes a little logical judgment in the background

The main content of the back end is: according to the MD5 value transmitted from the front end to the back end, search the server disk to see if there is any unfinished file merging information (namely unfinished semi-finished file slices), and return data to tell the front end which section to upload from according to the number of uploaded slices

If you want to suspend the upload of a slice, you can use the abort method of XMLHttpRequest

Three, use scenarios

  • Accelerated upload of large files: When the file size exceeds the expected size, sharding upload can be used to upload multiple parts in parallel to speed up the upload
  • If the network environment is poor, fragment upload is recommended. When an upload fails, only the failed Part needs to be retransmitted
  • Streaming upload: You can start uploading when the size of the file to be uploaded is uncertain. This scenario is common in video surveillance and other industries

summary

The current pseudo-code just provides a simple idea, but to make things as extreme as possible, we need to consider more scenarios, such as

  • What if the slice upload fails
  • How to refresh the page during upload
  • How to do parallel upload
  • When should slices be cut by quantity and by size
  • How to handle large file upload with Web Worker
  • How to achieve second transmission

How to implement pull up load, pull down refresh?

One, foreword

Pull-down refresh and pull-up load are two interactions that are common on mobile

Essentially the same as paging in a PC web page, but in a different form of interaction

The open source community also has many excellent solutions, such as iscroll, Better-Scroll, pulltorefresh. Js libraries, etc

These third-party libraries are very easy to use

We implement a pull-up load and pull-down refresh in a native way, which helps to have a better understanding and use of third-party libraries

Two, the implementation principle

Both pull-up loading and pull-down refreshing depend on user interaction

The most important thing to understand is in what context and when do interactions occur

Pull on loading

Let’s start with a picture

The essence of pull-up loading is when the page hits bottom, or is about to hit bottom

To determine the bottom of a page, we need to know the following attributes

  • scrollTop: Height distance of scroll windowwindowThe distance at the top, which increases as you scroll up, starts at 0, which is a variable value
  • clientHeight: is a fixed value that represents the height of the visible area of the screen;
  • scrollHeight: also exists when the page cannot scroll, when scrollHeight equals clientHeight. ScrollHeight saidbodyTotal length of all elements (including padding of the body element itself)

To sum up, we get a bottoming formula:

scrollTop + clientHeight >= scrollHeight
Copy the code

Simple implementation

let clientHeight  = document.documentElement.clientHeight; // Browser height
let scrollHeight = document.body.scrollHeight;
let scrollTop = document.documentElement.scrollTop;
 
let distance = 50;  // When the distance window is still 50, it starts to trigger;

if ((scrollTop + clientHeight) >= (scrollHeight - distance)) {
    console.log("Start loading data");
}
Copy the code

The drop-down refresh

The essence of a drop-down refresh is an action that needs to be triggered when the user pulls down the page at the top of the page itself

There are three main steps to the native implementation of the pull-down refresh:

  • Listen to nativetouchstartEvent that records the value of its initial position,e.touches[0].pageY;
  • Listen to nativetouchmoveEvent that records and calculates the difference between the current slide position value and the initial position value, greater than0Represents pulling down with the aid of CSS3translateYAttribute to slide the element down with the gesture, and should also set a maximum allowed to slide;
  • Listen to nativetouchendEvent that is triggered if the element slides to its maximum valuecallbackAnd at the same time willtranslateYReset to0, the element returns to its original position

Here’s an example:

The Html structure is as follows:

<main>
    <p class="refreshText"></p >
    <ul id="refreshContainer">
        <li>111</li>
        <li>222</li>
        <li>333</li>
        <li>444</li>
        <li>555</li>.</ul>
</main>
Copy the code

Listen for the TouchStart event to record the initial value

var _element = document.getElementById('refreshContainer'),
    _refreshText = document.querySelector('.refreshText'),
    _startPos = 0.// The initial value
    _transitionHeight = 0; // The distance moved

_element.addEventListener('touchstart'.function(e) {
    _startPos = e.touches[0].pageY; // Record the initial position
    _element.style.position = 'relative';
    _element.style.transition = 'transform 0s';
}, false);
Copy the code

Listen for the TouchMove movement event and record the slide difference


_element.addEventListener('touchmove'.function(e) {
    // e.touches[0].pagey
    _transitionHeight = e.touches[0].pageY - _startPos; // Record the difference

    if (_transitionHeight > 0 && _transitionHeight < 60) { 
        _refreshText.innerText = 'Pull refresh'; 
        _element.style.transform = 'translateY('+_transitionHeight+'px)';

        if (_transitionHeight > 55) {
            _refreshText.innerText = 'Release update'; }}},false);
Copy the code

And finally, listen for the TouchEnd to leave

_element.addEventListener('touchend'.function(e) {
    _element.style.transition = 'the transform 0.5 s ease 1 s';
    _element.style.transform = 'translateY(0px)';
    _refreshText.innerText = 'In update... ';
    // todo...

}, false);
Copy the code

As can be seen from the above, there are three stages in the pull-down to release process:

  • When the difference between the current gesture sliding position and the initial position is greater than zero, a drop-down refresh operation is prompted
  • When it reaches a certain value, the operation prompt after release will be displayed
  • When the drop down reaches the set maximum value, the callback is executed, indicating that an update operation is in progress

Three cases,

In actual development, we use more third-party libraries, and the following uses Better-Scroll as an example:

HTML structure

<div id="position-wrapper">
    <div>
        <p class="refresh">Drop down refresh </p ><div class="position-list">
   <! -- List contents -->
        </div>
        <p class="more">See more </p ></div>
</div>
Copy the code

Instantiate the pull-up plug-in and register the plug-in with use

import BScroll from "@better-scroll/core";
import PullDown from "@better-scroll/pull-down";
import PullUp from '@better-scroll/pull-up';
BScroll.use(PullDown);
BScroll.use(PullUp);
Copy the code

Instantiate BetterScroll and pass in the parameters

let pageNo = 1,pageSize = 10,dataList = [],isMore = true;  
var scroll= new BScroll("#position-wrapper", {scrollY:true.// Scroll vertically
    click:true.// The browser's native click event is blocked by default. If clicking is required, set this to true
    pullUpLoad:true.// Pull up to load more
    pullDownRefresh: {threshold:50.// The location of the pullingDown event
        stop:0// Drop down the position to stay after bouncing back}});// Listen for the pull-down refresh
scroll.on("pullingDown",pullingDownHandler);
// Monitor real-time scrolling
scroll.on("scroll",scrollHandler);
// Pull up to load more
scroll.on("pullingUp",pullingUpHandler);

async function pullingDownHandler(){
    dataList=[];
    pageNo=1;
    isMore=true;
    $(".more").text("See more");
    await getlist();// Request data
    scroll.finishPullDown();// Perform this operation after each pull-down
    scroll.refresh();// This operation is required when the dom structure of the scroll area changes
}
async function pullingUpHandler(){
    if(! isMore){ $(".more").text("There's no more data.");
        scroll.finishPullUp();// Perform this operation after each pull-up
        return;
    }
    pageNo++;
    await this.getlist();// Request data
    scroll.finishPullUp();// Perform this operation after each pull-up
    scroll.refresh();// This operation is required when the dom structure of the scroll area changes
}
function scrollHandler(){
    if(this.y>50The $()'.refresh').text("Let go and load.");
    else $('.refresh').text("Drop-down refresh");
}
function getlist(){
    // The data returned
    letresult=.... ; dataList=dataList.concat(result);// Check whether the load is complete
    if(result.length<pageSize) isMore=false;
    // Render the dataList to the HTML content
}    
Copy the code

Note:

Note the following points when using better Scroll to refresh and load:

  • wrapperMust have only one child element in
  • The height of the child element is greater thanwrapperhigher
  • Be sure to use itDOMWhether the element has been generated must wait untilDOMAfter rendering is complete, thennew BScroll()
  • Roll-zoneDOMWhen the structure of an element changes, a refresh is requiredrefresh()
  • After the drop-down or drop-down is complete, you need to execute the drop-downfinishPullUp()orfinishPullDown()Otherwise, the next operation will not be performed
  • better-scrollBy default, this prevents the browser from being nativeclickEvent, if you want to add a click event to the scroll content area, you need to set it in the instantiation propertyclick:true

summary

The principle of pull-down refresh and pull-up loading is very simple, but the real complexity is the compatibility, ease of use, performance and many other details to consider in the encapsulation process

What is single sign-on? How to do that?

What is it

Single Sign On, or SSO for short, is one of the more popular solutions for enterprise business integration

SSO is defined as one login in multiple applications that allows users to access all trusted applications

SSO generally requires an independent authentication authority (Passport), which is used to log in to subsystems. The subsystem itself does not participate in the login operation

When a system is successfully logged in, Passport will issue a token to each subsystem, which can obtain their own protected resources by holding the token. In order to reduce frequent authentication, each subsystem will establish a local session after being authorized by Passport, so it does not need to initiate authentication to Passport again within a certain period of time

In the figure above, there are four systems, namely Application1, Application2, Application3, and SSO. When Application1, Application2, and Application3 need to log in, the SSO system will complete the login. Other applications are logged in accordingly

For example

Both Taobao and Tmall belong to Ali. When a user logs in to Taobao and opens Tmall, the system automatically helps the user to log in to Tmall. This phenomenon is single sign-on

Two, how to achieve

Single sign-on under the same domain name

The domain property of the cookie is set to the parent domain of the current domain, and the cookies of the parent domain are shared by the domain. The path property defaults to the context path of the Web application

Taking advantage of this feature of Cookie, yes, we only need to set the domain attribute of Cookie to the domain name of the parent domain (master domain), set the path attribute of Cookie to the root path, and save the Session ID (or Token) to the parent domain. This Cookie is then accessible to all subdomain applications

However, this requires the domain name of the application system to be established under a common main domain name, such as Tieba.baidu.com and map.baidu.com, which are both established under the main domain name baidu.com, so they can realize single sign-on (SSO) in this way

Single Sign-on under Different Domain names (1)

In the case of different domains where cookies are not shared, we can deploy an authentication authority for a separate Web service dedicated to handling login requests

After successful login, the authentication center records the login status of the user and writes the token into the Cookie (note that the Cookie belongs to the authentication center and cannot be accessed by the application system).

The application system checks whether the current request has a Token. If the request does not have a Token, the user has not logged in to the current system and is redirected to the authentication center

Because this operation automatically brings the authentication authority’s Cookie, the authentication authority can know whether the user has logged in based on the Cookie

If the authentication center finds that the user has not logged in, it returns to the login page and waits for the user to log in

If the user has already logged in, the system will not allow the user to log in again. Instead, the system will redirect to the target URL and generate a Token after the target URL before the redirect and send it back to the target application system

After obtaining the Token, the application system needs to verify the validity of the Token with the authentication center to prevent users from forging the Token. After confirming the information, the application system records the login status of the user, writes the Token into a Cookie, and permits the access. When the user accesses the current application system again, the Token will be automatically worn by the user. The application system verifies the Token and finds that the user has logged in, so there will be no authentication center

This implementation method is relatively complex, supports cross-domain, and has good scalability. It is the standard practice of single sign-on (SSO)

Single Sign-on under Different Domain names (2)

You can choose to save the Session ID (or Token) in the LocalStorage of the browser, so that the front end proactively passes the LocalStorage data to the server each time the back end sends a request

All of this is controlled by the front end, and all the back end needs to do is pass the Session ID (or Token) in the response body to the front end after the user logs in successfully

Single sign-on is entirely possible on the front end. After receiving the Session ID (or Token), the front-end can write it to localstorages in other domains by special means in addition to its own LocalStorage

The key codes are as follows:

/ / access token
var token = result.data.token;
 
// Dynamically create an invisible IFrame and load a cross-domain HTML in the iframe
var iframe = document.createElement("iframe");
iframe.src = "http://app1.com/localstorage.html";
document.body.append(iframe);
// Pass the token to iframe using the postMessage() method
setTimeout(function () {
    iframe.contentWindow.postMessage(token, "http://app1.com");
}, 4000);
setTimeout(function () {
    iframe.remove();
}, 6000);
 
// Bind an event listener to the HTML loaded by the iframe. When the event is triggered, the received token data is written to localStorage
window.addEventListener('message'.function (event) {
    localStorage.setItem('token', event.data)
}, false);
Copy the code

The front-end uses iframe+postMessage() to write the same Token to LocalStorage in multiple domains. Each time before sending a request to the back-end, the front-end will actively read the Token from LocalStorage and carry it in the request. This enables the same Token to be shared by multiple domains

This implementation is completely controlled by the front end, requires little back-end involvement, and also supports cross-domain

Three, processes,

The flow chart of single sign-on is as follows:

  • When a user accesses the protected resource of system 1, system 1 finds that the user does not log in, switches to the SSO authentication center, and takes its own address as a parameter
  • The SSO authentication center detects that the user has not logged in and directs the user to the login page
  • The user enters the user name and password to submit a login application
  • The SSO authentication center verifies user information, creates a global session between the user and the SSO authentication center, and creates an authorization token
  • Sso authority redirects the original request address with the token (System 1)
  • System 1 obtains the token and goes to the SSO authentication center to verify whether the token is valid
  • The SSO authentication authority verifies the token, returns valid, and registers system 1
  • System 1 uses this token to create a session with the user, called a local session, that returns the protected resource
  • Users access protected resources in system 2
  • System 2 detects that the user does not log in, switches to the SSO authentication center, and sets its own address as a parameter
  • When the SSO authentication center detects that the user has logged in, the sso authentication center switches back to the address of system 2 with the token attached
  • System 2 obtains the token and goes to the SSO authentication center to verify whether the token is valid
  • Sso authentication center validates token, return valid, register system 2
  • System 2 uses this token to create a local session with the user and return the protected resource

After a successful login, a user establishes sessions with the SSO authentication center and each subsystem. The sessions established between the user and the SSO authentication center are called global sessions

Sessions established between users and subsystems are called local sessions. After a local session is established, users do not access the subsystem protected resources through the SSO authentication center

Global sessions and local sessions have the following constraints:

  • If a local session exists, a global session must exist
  • Global sessions exist, but local sessions do not necessarily exist
  • Global session destruction, local session must be destroyed