Node file breakpoint continuation principle and method summary

Introduction: I have done a small project before, which used the file upload, in the large file on the use of breakpoint continuation, reduce the pressure on the server, now do a detailed summary of this development experience.

The principle is introduced

Here is the principle of file upload to help clear up the confusion.

Ordinary upload

There are many ordinary uploads on general websites, most of which are uploading some users’ profile pictures and users’ dynamic comments with pictures, so let’s talk about the principle of this first.

After the user selects the file, JS detects whether the file size exceeds the limit and whether the file format is correct.
After checking, ajax is used to submit the request, and the server also carries out secondary verification and stores it to the server.
The back end returns the file address to the front end, rendering the page data;

Large file upload

After the user selects the file, JS detects whether the file size exceeds the limit and whether the file format is correct.
Based on the file size, usefile.sliceMethods File segmentation;
useSparkMD5andFileReaderAPI generates a unique MD5 value for the file;
Use Ajax to submit the request, the server receives the file back on the server information;
- If the md5 value exists, returns the number of files uploaded in the folder.
- If it does not exist, create a new folder with the MD5 value and return empty contents.
After receiving the information, the front end makes a judgment according to the returned information;
- If the returned file length is equal to the total length of the slice, the file is requested to be merged;
- If the returned file length is less than the total length of the slice, the corresponding slice file will be uploaded until the last slice is uploaded and then the file will be merged.
After receiving the merge request, the back end merges the files in the folder with the corresponding MD5 value and returns the file address.
After receiving the file address, the front end renders the page data;

Breakpoint continuingly

This means that in the process of file uploading, if there is force majeure, such as network interruption, server exception, or other reasons, the upload is interrupted;

When uploading the file again, the server finds out how many files are uploaded and how many files are unuploaded according to the MD5 value of the file, and sends the file to the client. After receiving the file, the client continues to upload the unuploaded files, merges the files and returns the address.

In this way, it avoids the repeated uploading of files, which wastes server space and saves server resources. Moreover, it is faster and more efficient than uploading a large file.

Methods to summarize

Next, according to the above logic principle analysis steps, code function realization.

Common file

This section describes uploading common files, including the front end and the back end.

The front part

HTML part

Let’s start by building a small house

<div class="upload">
        <h3>Ordinary upload</h3>
        <form>
            <div class="upload-file">
                <label for="file">Please select file</label>
                <input type="file" name="file" id="file" accept="image/*">
            </div>
            <div class="upload-progress">Current Progress:<p>
                    <span style="width: 0;" id="current"></span>
                </p>
            </div>
            <div class="upload-link">File Address:<a id="links" href="javascript:void();" target="_blank">The file link</a>
            </div>
        </form>
    </div>
    <div class="upload">
        <h3>Large file upload</h3>
        <form>
            <div class="upload-file">
                <label for="file">Please select file</label>
                <input type="file" name="file" id="big-file" accept="application/*">
            </div>
            <div class="upload-progress">Current Progress:<p>
                    <span style="width: 0;" id="big-current"></span>
                </p>
            </div>
            <div class="upload-link">File Address:<a id="big-links" href="" target="_blank">The file link</a>
            </div>
        </form>
    </div>
Copy the code

Two JS files axios and spark-MD5 are introduced.

<script src="https://cdn.bootcdn.net/ajax/libs/axios/0.21.1/axios.min.js"></script>
<script src="https://cdn.bootcdn.net/ajax/libs/spark-md5/3.0.0/spark-md5.min.js"></script>
Copy the code

The CSS part

Let’s decorate the house.

body {
    margin: 0;
    font-size: 16px;
    background: #f8f8f8;
}
h1.h2.h3.h4.h5.h6.p {
    margin: 0;
}

/* * { outline: 1px solid pink; } * /

.upload {
    box-sizing: border-box;
    margin: 30px auto;
    padding: 15px 20px;
    width: 500px;
    height: auto;
    border-radius: 15px;
    background: #fff;
}

.upload h3 {
    font-size: 20px;
    line-height: 2;
    text-align: center;
}

.upload .upload-file {
    position: relative;
    margin: 30px auto;
}

.upload .upload-file label {
    display: flex;
    justify-content: center;
    align-items: center;
    width: 100%;
    height: 150px;
    border: 1px dashed #ccc;
}

.upload .upload-file input {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
    opacity: 0;
}

.upload-progress {
    display: flex;
    align-items: center;
}

.upload-progress p {
    position: relative;
    display: inline-block;
    flex: 1;
    height: 15px;
    border-radius: 10px;
    background: #ccc;
    overflow: hidden;
}

.upload-progress p span {
    position: absolute;
    left: 0;
    top: 0;
    width: 0;
    height: 100%;
    background: linear-gradient(to right bottom, rgb(163.76.76), rgb(231.73.52));
    transition: all .4s;
}

.upload-link {
    margin: 30px auto;
}

.upload-link a {
    text-decoration: none;
    color: rgb(6.102.192);
}

@media all and (max-width: 768px) {
    .upload {
        width: 300px; }}Copy the code

Js part

Finally, add interactive effects.

// Get the element
const file = document.querySelector('#file');
let current = document.querySelector('#current');
let links = document.querySelector('#links');
let baseUrl = 'http://localhost:3000';

// Listen for file events
file.addEventListener('change'.(e) = > {
    console.log(e.target.files);
    let file = e.target.files[0];
    if (file.type.indexOf('image') = = -1) {
        return alert('File format can only be pictures! ');
    }
    if ((file.size / 1000) > 100) {
        return alert('Files cannot be larger than 100KB! ');
    }
    links.href = ' ';
    file.value = ' ';
    this.upload(file);
}, false);

// Upload the file
async function upload (file) {
    let formData = new FormData();
    formData.append('file', file);
    let data = await axios({
        url: baseUrl+'/upload'.method: 'post'.data: formData,
        onUploadProgress: function(progressEvent) {
            current.style.width = Math.round(progressEvent.loaded / progressEvent.total * 100) + The '%'; }});if (data.data.code == 200) {
        links.href = data.data.data.url;
    } else {
        alert('Upload failed! ')}}Copy the code

The backend part

Open the last folder demo, download and install a file processing package formidable, and then start processing uploaded files!

Create a new folder and don’t forget to introduce new files in app.js.

const upload = require('./upload/index');

app.use(express.static(path.join(__dirname, 'public')));
app.use('/file', express.static(path.join(__dirname, 'static')));

app.use('/upload', upload);
Copy the code

The following is a file hierarchy diagram.

-- static
-- big -- doc -- temp -- upload - index.js - util.js -- app.jsCopy the code

const express = require('express');
const Router = express.Router();
const formidable = require('formidable');
const path = require('path');
const fs = require('fs');
const baseUrl = 'http://localhost:3000/file/doc/';
const dirPath = path.join(__dirname, '.. /static/')

// Upload ordinary files
Router.post('/'.(req, res) = > {
    let form = formidable({
        multiples: true.uploadDir: dirPath+'temp/'
    })

    form.parse(req, (err,fields, files) = > {
        if (err) {
            return res.json(err);
        }
        let newPath = dirPath+'doc/'+files.file.name;
        fs.rename(files.file.path, newPath, function(err) {
            if (err) {
                return res.json(err);
            }
            return res.json({
                code: 200.msg: 'get_succ'.data: {
                    url: baseUrl + files.file.name
                }
            })
        })
        
    })

});

module.exports = Router;
Copy the code

A large file

This large file breakpoint continuation, in fact, is in the previous file upload on the basis of further extension. So the structure and style of the front end are the same, but the method is different.

The front part

Here is a method introduction.

Access to elements

const bigFile = document.querySelector('#big-file');
let bigCurrent = document.querySelector('#big-current');
let bigLinks = document.querySelector('#big-links');
let fileArr = [];
let md5Val = ' ';
let ext = ' ';
Copy the code

Test file

bigFile.addEventListener('change'.(e) = > {
    let file = e.target.files[0];
    if (file.type.indexOf('application') = = -1) {
        return alert('File format can only be document application! ');
    }
    if ((file.size / (1000*1000)) > 100) {
        return alert('Files cannot be larger than 100MB! ');
    }
    this.uploadBig(file);
}, false);
Copy the code

The cutting file

// Cut files
function sliceFile (file) {
    const files = [];
    const chunkSize = 128*1024;
    for (let i = 0; i < file.size; i+=chunkSize) {
        const end = i + chunkSize >= file.size ? file.size : i + chunkSize;
        let currentFile = file.slice(i, (end > file.size ? file.size : end));
        files.push(currentFile);
    }
    return files;
}
Copy the code

Obtain the MD5 value of the file

// Get the MD5 value of the file
function md5File (files) {
    const spark = new SparkMD5.ArrayBuffer();
    let fileReader;
    for (var i = 0; i < files.length; i++) {
        fileReader = new FileReader();
        fileReader.readAsArrayBuffer(files[i]);
    }
    return new Promise((resolve) = > {
        fileReader.onload = function(e) {
            spark.append(e.target.result);
            if(i == files.length) { resolve(spark.end()); }}})}Copy the code

Uploading fragmented Files

async function uploadSlice (chunkIndex = 0) {
    let formData = new FormData();
    formData.append('file', fileArr[chunkIndex]);
    let data = await axios({
        url: `${baseUrl}/upload/big? type=upload&current=${chunkIndex}&md5Val=${md5Val}&total=${fileArr.length}`.method: 'post'.data: formData,
    })

    if (data.data.code == 200) {
        if (chunkIndex < fileArr.length -1 ){
            bigCurrent.style.width = Math.round((chunkIndex+1) / fileArr.length * 100) + The '%';
            ++chunkIndex;
            uploadSlice(chunkIndex);
        } else{ mergeFile(); }}}Copy the code

Merge files

async function mergeFile () {
    let data = await axios.post(`${baseUrl}/upload/big? type=merge&md5Val=${md5Val}&total=${fileArr.length}&ext=${ext}`);
    if (data.data.code == 200) {
        alert('Upload successful! ');
        bigCurrent.style.width = '100%';
        bigLinks.href = data.data.data.url;
    } else{ alert(data.data.data.info); }}Copy the code

The backend part

To obtain parameters

Get upload parameters and make a judgment.

let type = req.query.type;
let md5Val = req.query.md5Val;
let total = req.query.total;
let bigDir = dirPath + 'big/';
let typeArr = ['check'.'upload'.'merge'];
if(! type) {return res.json({
        code: 101.msg: 'get_fail'.data: {
            info: 'Upload type cannot be empty! '}})}if(! md5Val) {return res.json({
        code: 101.msg: 'get_fail'.data: {
            info: 'File MD5 value cannot be empty! '}})}if(! total) {return res.json({
        code: 101.msg: 'get_fail'.data: {
            info: 'File slice number cannot be empty! '}})}if(! typeArr.includes(type)) {return res.json({
        code: 101.msg: 'get_fail'.data: {
            info: 'Upload type error! '}})}Copy the code

The type is detection

let filePath = `${bigDir}${md5Val}`;
fs.readdir(filePath, (err, data) = > {
    if (err) {
        fs.mkdir(filePath, (err) = > {
            if (err) {
                return res.json({
                    code: 101.msg: 'get_fail'.data: {
                        info: 'Fetch failed! ',
                        err
                    }
                })
            } else {
                return res.json({
                    code: 200.msg: 'get_succ'.data: {
                        info: 'Success! '.data: {
                            type: 'write'.chunk: [].total: 0}}})}})}else {
        return res.json({
            code: 200.msg: 'get_succ'.data: {
                info: 'Success! '.data: {
                    type: 'read'.chunk: data,
                    total: data.length
                }
            }
        })
    }
    
})
Copy the code

The type is uploaded

let current = req.query.current;
if(! current) {return res.json({
        code: 101.msg: 'get_fail'.data: {
            info: 'File current shard value cannot be empty! '}})}let form = formidable({
    multiples: true.uploadDir: `${dirPath}big/${md5Val}/ `,
})

form.parse(req, (err,fields, files) = > {
    if (err) {
        return res.json(err);
    }
    let newPath = `${dirPath}big/${md5Val}/${current}`;
    fs.rename(files.file.path, newPath, function(err) {
        if (err) {
            return res.json(err);
        }
        return res.json({
            code: 200.msg: 'get_succ'.data: {
                info: 'upload success! '}})})})Copy the code

Types are merged

let ext = req.query.ext;
if(! ext) {return res.json({
        code: 101.msg: 'get_fail'.data: {
            info: 'File suffixes cannot be empty! '}})}let oldPath = `${dirPath}big/${md5Val}`;
let newPath = `${dirPath}doc/${md5Val}.${ext}`;
let data = await mergeFile(oldPath, newPath);
if (data.code == 200) {
    return res.json({
        code: 200.msg: 'get_succ'.data: {
            info: 'File merge successful! '.url: `${baseUrl}${md5Val}.${ext}`}})}else {
    return res.json({
        code: 101.msg: 'get_fail'.data: {
            info: 'File merge failed! '.err: data.data.error
        }
    })
}
Copy the code

The createWriteStream and createReadStream methods of FS are used to implement the merge function.

Merge files

const fs = require('fs');

function mergeFile (filePath, newPath) {
    return new Promise((resolve, reject) = > {
        let files = fs.readdirSync(filePath),
        newFile = fs.createWriteStream(newPath);
        let filesArr = arrSort(files).reverse();
        main();
        function main (index = 0) {
            let currentFile = filePath + '/'+filesArr[index];
            let stream = fs.createReadStream(currentFile);
            stream.pipe(newFile, {end: false});
            stream.on('end'.function () {
                if (index < filesArr.length - 1) {
                    index++;
                    main(index);
                } else {
                    resolve({code: 200});
                }
            })
            stream.on('error'.function (error) {  
                reject({code: 102.data:{error}})
            })
        }
    })
}
Copy the code

File sorting

function arrSort (arr) {
    for (let i = 0; i < arr.length; i++) {
        for (let j = 0; j < arr.length; j++) {
            if (Number(arr[i]) >= Number(arr[j])) {
                lett = arr[i]; arr[i] = arr[j]; arr[j] = t; }}}return arr;
}
Copy the code

We practice

Now that I’ve written my method, let’s test it out.

Two files are prepared to test the two functions separately.

Common file

This is the normal file upload interface

After successful upload:

The backend returns:

Open file address preview:

You can see it!

A large file

This is the big file file upload interface

After successful upload:

This is a shard file being uploaded:

This is the file shard upload after merging the contents of the return:

Open file address preview:

Upload again found soon return file address:

This is a screenshot of the nodejs directory. You can see that the shards of the files are well preserved and well merged.

So much for file upload and breakpoint continuation. Of course, the method I mentioned above is just a reference. If you have a better way, please contact me.

Node file breakpoint continuation principle and method summary

directory

The principle is introduced

Ordinary upload

Large file upload

Breakpoint continuingly

Methods to summarize

Common file

The front part

The backend part

A large file

The front part

The backend part

We practice

Related Posts

Advanced usage of TypeScript

Stupid bird learning Diary Express framework introduction

Hexo + GitHub Actions + OSS blog system construction