Large file uploads and resumable uploads using browser local storage

The basic idea

Shard to upload

SessionStorage: No Blob is recorded in sessionStorage. Onload event is triggered after XMLHttpRequest upload is complete. Record the Blob just completed in sessionStorage.

Resumable breakpoint is the pause function

Continuing above, save the XMLHttpRequest object in an executing task container during an asynchronous XMLHttpRequest upload, and remove it from the task container when the ONLoad event is triggered after the XMLHttpRequest upload completes. When you want to suspend a file upload, you simply abort each XMLHttpRequest object in the task container. Restarting is a re-sharding upload, which skips the uploaded portion, so you can continue uploading at the same pace.

Front knowledge

Web Storage

LocalStorage and SessionStorage is one of the solutions provided by HTML5 browser localization storage, storage capacity can reach 5MB, easy to use. They share the same API, but differ in scope and lifecycle. If you do not manually delete the LocalStorage, it will be permanently stored in the LocalStorage. If the protocol, host name, and port on the page are the same, the same LocalStorage is read. SessionStorage will be cleared when the page window is closed. Of course, closing the browser will also clear the data, when the page window, protocol, host name, port is the same when sharing the same SessionStorage

Storage stores data in the form of keys and values as strings. Therefore, the Storage content is limited. You can use json. stringfy and json. parse to convert data.

API interface

sessionStorage/localStorage.
                            setItem(key, value)       // Add data
                            getItem(key)              // Find data
                            removeItem(key)           // Remove data
                            clear                     // Clear the data
Copy the code

The File object

The File object is a File property obtained after is read from a File. This property is a FileList object containing a File element. The File object is a special Blob object, which means that it can use any property and method of a Blob, and can be used in any bloB-type context, such as FileReader, xmlHttprequest.send ().

The File object is more like a File descriptor in the case of selecting a File from the local hard disk. Its name makes it easy to think that the File object has read the File from the hard disk, which it does not. To read the File into the browser’s memory, use the FileReader object. When we send asynchronously, we don’t have to read the File into the browser’s memory and then send it, xmlHttprequet.send handles the File object, and when we send it, the browser sends the actual File.

Common properties and methods

File.
     name            // Returns the name of the reference file
     size            // Returns the file size in bytes
     type            // Return the file Type as MIME Type
     slice(start, end, contentType)     // Split file, three parameters are optional, respectively, the file start position, the file end position, the new file type.
Copy the code

FormData object

FormData can be constructed as a key/value pair representing form data. The key does not need to be unique, but the data is sent using XMLHttpRequest. If the encoding type is set to multipart/form-data, it uses the same format as the form. It is not used in the code because the file receiving framework on the server side adds this encoding type to req by default. The reason for not sending File directly is to send with some other information, such as File number, verification code, etc.

API interface

new FormData().
               append(name, value [,filename])      // Add data. If value is of blob type, filename can be used to specify the name of the bloB
               delete(name)                         // All values of the key will be deleted
               entries()                            // Returns an iterable containing all key-value pairs
               keys()                               // Return an iterable containing all the keys
               values()                             // Returns an iterable containing all values
               get(name)                            // Returns the first value associated with the given key
               getAll(name)                         // Returns all values associated with the given key
               set(name, value)                     // If the key exists, the original value is overwritten. If the key does not exist, the new value is added
Copy the code

Code implementation

Simple server-side test code

const express = require('express');
const multer = require('multer');

const storage = multer.diskStorage({
    destination: function(req, file, cb) {
        cb(null.'./uploads/')},filename: function ( req, file, cb) {
        cb(null, req.body.name)
    }
})
const upload = multer({ storage })
const app = express();
const port = 5000;

app.use(express.json()) // for parsing application/json
app.use(express.urlencoded({ extended: true })) // for parsing application/x-www-form-urlencoded

app.all(The '*'.function(req, res, next) {
    res.header("Access-Control-Allow-Origin"."*");
    res.header("Access-Control-Allow-Headers"."X-Requested-With");
    res.header("Access-Control-Allow-Methods"."PUT,POST,GET,DELETE,OPTIONS");
    next();
});

app.post('/', upload.any(), (req, res) = > {
    res.header("Access-Control-Allow-Origin"."*");
    res.header("Access-Control-Allow-Headers"."*");
    res.header('Access-Control-Allow-Methods'.'PUT, GET, POST, DELETE, OPTIONS');
    console.log(req.files);
    res.send('Ok');
})

app.listen(port, () = > {
    console.log(`Example app listening at http://localhost:${port}`)})Copy the code

The front-end code

Page section

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta  name="viewport" content="width=device-width, Initial-scale =1.0"> <script SRC ="./upload.js"></script> <title> File upload test </title> </head> <body> <div class="upload-container"> <input type="file" multiple id="filename" name="file"/> <span id="progress">0% &nbsp; </span> <button onclick="submit()" </button> <button id='pause'> <script> var file; function submit() { file = document.getElementById('filename').files[0] file = new webUploader(file, 'http://localhost:5000'); document.getElementById('pause').addEventListener('click', file.pause.bind(file)); document.getElementById('cancel').addEventListener('click', file.cancel.bind(file)); } </script> </body> </html>Copy the code

The upload. Js code

function webUploader(file, url, headers={}){
    this._tasks = this._init(file);   // Split large files
    this._exeQueue = {};  // Save the executing XHR object for abort to use
    this._status = 'running'; // Uploader status description
    this._url = url;
    this._headers = headers;
    this._size = {totalSize: file.size, loadedSize: 0};

    document.getElementById('progress').innerHTML = '0%' + '  ';
    this._run();    // Start uploading
}


// Cut large files
webUploader.prototype._init = function (file) {
    // Returns the iterable of file cutting
    let tasks = [];
    let index = 0;
    const SIZE = 20 * 1024 * 1024;
    while(index < file.size) {
        tasks.push({name: file.name + '_' + (index / SIZE + 1), file: file.slice(index, index + SIZE)});
        index += SIZE;
    }
    return tasks;
}

// Upload
webUploader.prototype._run = async function () {
    let queue = [];
    this._exeQueue = {};
    for(task of this._tasks) {
        // Upload records have been retrieved locally, then skip
        if(! sessionStorage.getItem(task.name) || sessionStorage.getItem(task.name)! ='done') {
            let formData = new FormData();
            formData.append('name', task.name);
            formData.append('file', task.file);
            queue.push(this._request({url: this._url, formData: formData, headers: this._headers, exeQueue: this._exeQueue})); }}Promise.all(queue).then( value= > {
        console.log('Upload completed');
    }).catch (value= > {
        console.log('Upload failed');
    });
}

// Pause the function
webUploader.prototype.pause = function() {
    if ( this._status == 'running') {
        this._status = 'waiting';
        for(key in this._exeQueue) {
            this._exeQueue[key].abort();
        }
        this._exeQueue = {};
        console.log('Suspended');
    } else if( this._status == 'waiting') {
        this._status = 'running';
        console.log('Restarted');
        this._run();
    } else{}}// Cancel upload
webUploader.prototype.cancel = function () {
    if(this._status ! ='ending') {this._status = 'running';
        this.pause();
        this._status = 'ending';
        this._tasks.forEach( el= > {
            sessionStorage.removeItem(el.name);
        });
        this._tasks = [];
        this._url = ' ';
        this._headers = {};
        this._size = {totalSize: undefined.loadedSize: undefined};
        document.getElementById('progress').innerHTML = '0%' + '  ';
        console.log('Cancelled'); }}// Simply encapsulate asynchronous requests
webUploader.prototype._request = function({url, formData, headers, exeQueue}) {
    return new Promise( (resolve, reject) = > {
        let name = formData.get('name')
        xhr = new XMLHttpRequest();
        xhr.upload.onprogress = this._progressUpdate(this._size);
        xhr.open('post', url);
        Object.keys(headers).forEach( key= > {
            xhr.setRequestHeader(key, headers[key])
        });
        xhr.send(formData);
        xhr.onload = e= > {
            // Use sessionStore to store task information after uploading
            sessionStorage.setItem(name, 'done');
            resolve(
                {
                    data: e.target.response
                }
            );
            delete exeQueue[name]; // Delete the XHR from the upload
        }
        xhr.onerror = e= > {
            reject('error') } exeQueue[name] = xhr; })}// Update progress bar usage
webUploader.prototype._progressUpdate = function (size) {
    let oldloaded = 0;
    return e= > {
        let progress_dom = document.getElementById('progress');     // The test is directly obtained, simpler
        size.loadedSize += (e.loaded-oldloaded);
        oldloaded = e.loaded;
        progress_dom.innerHTML = Math.floor(( size.loadedSize / size.totalSize) * 100) + The '%' + '  '; }}Copy the code

Refer to the article

Bytedance Interviewer: Please implement a large file upload and resumable breakpoint

LocalStorage will know will know