preface

Ordinary business requirements: upload pictures, Excel, etc., after all, the size of a few meters can be quickly uploaded to the server. For uploaded videos and other large files with a size of hundreds of meters or a few GIGABytes, it takes a long time to wait. This has produced the corresponding solution, for the suspension of large file upload, disconnection, poor network, using slice + breakpoint continuation can be a good response to the above situation

Project analysis

  1. slice
    • That is, the uploaded video is segmented as follows:
    • File.slice(start,end) : Returns a new BLOb object
      • Copy the start byte of the bloB
      • Copy the end byte of the BLOB
  2. Breakpoint continuingly
    • Before uploading slices each time, request the server interface to read the uploaded slices of the same file
    • If a new file is uploaded, the server returns 0; otherwise, the number of uploaded slices is returned

Specific solution process

The demo provides key ideas and methods, other functions such as: file limit, lastModifiedDate check file repeatability, cache file periodically cleaning function extension can be added to this code base.

  • html
<input class="video" type="file" />
<button type="submit" onclick="handleVideo(event, '.video', 'video')">submit</button>
Copy the code
  • script
let count = 0; // Record the subscript of the file to be uploaded
const handleVideo = async (event, name, url) => {
// Block browser default form events
event.preventDefault();
let currentSize = document.querySelector("h2");
let files = document.querySelector(name).files;
// Default number of slices
const sectionLength = 100;
// First request the interface to see if the file exists on the server
// If count is 0, the file is uploaded for the first time. If count is not 0, the file exists on the server and the number of uploaded slices is returned
count = await handleCancel(files[0]);

// Declare an array object to hold slices
let fileCurrent = [];
// Loop file objects
for (const file of [...files]) {
  // Get the size of each slice
  let itemSize = Math.ceil(file.size / sectionLength);
  // loop file size, file blob into array
  let current = 0;
  for (current; current < file.size; current += itemSize) {
    fileCurrent.push({ file: file.slice(current, current + itemSize) });
  }
  // Axios simulates a manual cancellation request
  const CancelToken = axios.CancelToken;
  const source = CancelToken.source();
  // When the breakpoint is resumed, the number of slices is processed. The uploaded slices do not need to be uploaded again
  fileCurrent =
    count === 0 ? fileCurrent : fileCurrent.slice(count, sectionLength);
  // Loop slice request interface
  for (const [index, item] of fileCurrent.entries()) {
    / / simulation request suspend | | network disconnection
    if (index > 90) {
      source.cancel("Cancel request");
    }
    // Store file related information
    // file is a sliced blob object
    // filename is a filename
    // index is the current number of slices
    // total indicates the total number of slices
    let formData = new FormData();
    formData.append("file", item.file);
    formData.append("filename", file.name);
    formData.append("total", sectionLength);
    formData.append("index", index + count + 1);

    await axios({
      url: `http://localhost:8080/${url}`.method: "POST".data: formData,
      cancelToken: source.token,
    })
      .then((response) = > {
        // Return data to show progress
        currentSize.innerHTML = ` progress${response.data.size}% `;
      })
      .catch((err) = > {
        console.log(err); }); }}};// Request the interface to check whether the uploaded file exists
// If count is 0, it does not exist. If count is not 0, the corresponding slices have been uploaded
const handleCancel = (file) = > {
return axios({
  method: "post".url: "http://localhost:8080/getSize".headers: { "Content-Type": "application/json; charset = utf-8" },
  data: {
    fileName: file.name,
  },
})
  .then((res) = > {
    return res.data.count;
  })
  .catch((err) = > {
    console.log(err);
  });
};
Copy the code
  • The node server
// Build the server API with Express
const express = require("express");
// Import file upload logic code
const upload = require("./upload_file");
// Process all responses, set cross-domain
app.all("*".(req, res, next) = > {
  res.header("Access-Control-Allow-Origin"."*");
  res.header("Access-Control-Allow-Headers"."X-Requested-With");
  res.header("Access-Control-Allow-Methods"."PUT,POST,GET,DELETE,OPTIONS");
  res.header("Access-Control-Allow-Headers"."Content-Type, X-Requested-With ");
  res.header("X-Powered-By".3.2.1 "");
  res.header("Content-Type"."application/json; charset=utf-8");
  next();
});
const app = express();

app.use(bodyParser.json({ type: "application/*+json" }));
// Video upload (query current slice number)
app.post("/getSize", upload.getSize);
// Video upload interface
app.post("/video", upload.video);

// Enable local port listening
app.listen(8080);
Copy the code
  • upload_file
// File upload module
const formidable = require("formidable");
// File system module
const fs = require("fs");
// System path module
const path = require("path");

// Writes to the file stream
const handleStream = (item, writeStream) = > {
  // Read the corresponding directory file buffer
  const readFile = fs.readFileSync(item);
  / / will read buffer | | the chunk written to the stream
  writeStream.write(readFile);
  // After writing, clear the temporary slice file
  fs.unlink(item, () = > {});
};

// Upload video (slice)
module.exports.video = (req, res) = > {
  // Create a parse object
  const form = new formidable.IncomingForm();
  // Set the upload path for the video file
  let dirPath = path.join(__dirname, "video");
  form.uploadDir = dirPath;
  // Whether to retain the suffix of the upload file name
  form.keepExtensions = true;
  // Err error objects contain error messages if parsing fails
  // Fields contains key-value objects for formData other than binary
  // file Object type Information about the uploaded file
  form.parse(req, async (err, fields, file) => {
    // Get the blob object of the uploaded file
    let files = file.file;
    // Get the current slice index
    let index = fields.index;
    // Get the total number of slices
    let total = fields.total;
    // Get the file name
    let filename = fields.filename;
    // Rewrite the upload file name and set the temporary directory
    let url =
      dirPath +
      "/" +
      filename.split(".") [0] +
      ` _${index}. ` +
      filename.split(".") [1];
    try {
      // Change the upload file name
      fs.renameSync(files.path, url);
      console.log(url);
      // Asynchronous processing
      setTimeout(() = > {
        // Determine whether the last slice is uploaded, splice and write all videos
        if (index === total) {
          // Create a new directory to store the entire video
          let newDir = __dirname + `/uploadFiles/The ${Date.now()}`;
          // Create directory
          fs.mkdirSync(newDir);
          // Create a writable stream that can be used to write files
          let writeStream = fs.createWriteStream(newDir + ` /${filename}`);
          let fsList = [];
          // Take out all the slice files and put them in the array
          for (let i = 0; i < total; i++) {
            const fsUrl =
              dirPath +
              "/" +
              filename.split(".") [0] +
              ` _${i + 1}. ` +
              filename.split(".") [1];
            fsList.push(fsUrl);
          }
          // Loop slice file array to write stream
          for (let item of fsList) {
            handleStream(item, writeStream);
          }
          // Close stream to write to the streamwriteStream.end(); }},100);
    } catch (e) {
      console.log(e);
    }
    res.send({
      code: 0.msg: "Upload successful".size: index,
    });
  });
};

// Get the number of file slices
module.exports.getSize = (req, res) = > {
  let count = 0;
  req.setEncoding("utf8");
  req.on("data".function (data) {
    let name = JSON.parse(data);
    let dirPath = path.join(__dirname, "video");
    // Calculate the number of uploaded slice files
    let files = fs.readdirSync(dirPath);
    files.forEach((item, index) = > {
      let url =
        name.fileName.split(".") [0] +
        ` _${index + 1}. ` +
        name.fileName.split(".") [1];
      if(files.includes(url)) { ++count; }}); res.send({code: 0.msg: "Please continue uploading.",
      count,
    });
  });
};
Copy the code

Logical analysis

  • The front end
    • First, upload a file to check whether the file is uploaded for the first time or a corresponding slice already exists
      • When the file is uploaded for the first time, the slice starts from 0
      • If the file already has corresponding slices, upload the file from the number of slices
    • Circular slice array, upload each slice file
      • An analog manual pause request is used to cancel the request when the number of slices is greater than 90
  • The service side
    • The SVN receives a query file named filename, searches for the address of the temporary file, and checks whether the corresponding uploaded file exists
      • If this file has never been uploaded, 0 is returned and the number of slices starts from 0
      • If files have been uploaded, the number of slices is returned
    • Receive slices of uploaded files and store them in temporary storage directory
      • Check whether the slice is uploaded by count and total
      • After the upload is complete, create a directory to save the file and create a writable stream for the write operation
      • Extract the corresponding temporary files into the array, loop through the file directory array, read and write the file buffer in turn
      • After writing, close the writable stream.

summary

This is just one of the ways in which the code above can be implemented as it relates to specific business processes that may change or deviate. Hope this article can be helpful to you, if there are wrong places to write also hope to give advice or two.

  • Above code address: github.com/Surprise-li…

Past oliver

GitHook Tools — Husky (Formatting Code)

Browser Thread – Event Loop

JavaScript Event Delegate

Box-sizing: Box Model