preface
Recently, I am learning the knowledge related to uploading large files, and sum up the knowledge I have learned.
Key points:
- Confirm file format by binary information
- Three methods are used to compute file hash
- Section to upload
- Second transmission and breakpoint continuation
- Concurrency control and error retries
1. Project construction
1.1 Run commands to initialize the project
npx create-nuxt-app nuxt-demo
Copy the code
After the installation, go to the project directory CD nuxt-demo
1.2 Dependencies required by the installation project
NPM I stylus [email protected] koa-body koa- Router FS - Extra Spark-MD5Copy the codeStylus is used for style processing
Koa-body is used to get the body data of the form
Koa-router Provides routes
Fs-extra is an extension of FS and provides many convenient file manipulation methods
Spark-md5 is used to calculate MD5
1.3 axios configuration
Create axios.js in the plugins folder
// plugins/axios.js import Vue from 'Vue' import axios from 'axios' let service = axios.create({// prefix baseURL:'/ API '}) Vue.prototype.$http =service export const http = serviceCopy the codeConfigure the AXIos plug-in in nuxt.config.js
.plugins: [
'@/plugins/element-ui'.'@/plugins/axios'].Copy the code1.4 configuration koa
Add upload.js to server folder to prompt file upload interface
// server/upload.js
const KoaRouter = require('koa-router')
const router = new KoaRouter({
prefix: '/api'
})
router.get('/hello'.ctx= > {
ctx.body = 'hello koa'
})
module.exports = router
Copy the codeIntroduce koa-body and upload.js in server/index.js
.const { Nuxt, Builder } = require('nuxt')
const KoaBody = require('koa-body')
const uploadInterface = require('./upload')
app.use(KoaBody({ multipart: true }));
app.use(uploadInterface.routes()).use(uploadInterface.allowedMethods())
app.use((ctx) = > {
ctx.status = 200
ctx.respond = false // Bypass Koa's built-in response handling
ctx.req.ctx = ctx // This might be useful later on, e.g. in nuxtServerInit or with nuxt-stash
nuxt.render(ctx.req, ctx.res)
})
Copy the codeStart the project
npm run dev
Copy the codeAt this moment, visit http://localhost:3000/api/hello, if show hello koa koa configuration indicates success.
2. Implement simple file upload
Create a new upload.vue on page
// page/upload.vue <template> <div> <div id="drag"> <input type="file" name="file" @change="handleFileChange" /> </div> <div> <el-progress :stroke-width="20" :text-inside="true" :percentage="uploadProgress" ></el-progress> </div> <div> <el-button @click="uploadFile"> </el-button> </div> </template> <script> export default {data(){return{ uploadProgress:0 } }, methods:{ handleFileChange(e){ this.file = e.target.files[0] }, async uploadFile(){ if(! This.file){return} // Create formData const formData = new formData () formdata.append ('file', $http.post('/uploadfile') formData.append('name', this.file.name) const ret = await this. formData) console.log(ret); } } } </script> <style lang="stylus"> #drag height 100px line-height 100px border 2px dashed #eee text-align center </style>Copy the codeAdd a uploadfile route to server/upload.js
const KoaRouter = require('koa-router')
const path = require('path')
const fse = require('fs-extra')
const UPLOAD_DIR = path.resolve(__dirname, '.. /static/upload')
const router = new KoaRouter({
prefix: '/api'
})
router.get('/hello'.ctx= > {
ctx.body = 'hello koa'
})
router.post('/uploadfile'.ctx= > {
const { name } = ctx.request.body
const file = ctx.request.files.file
// Check whether the file exists. If not, copy the temporary file to the directory where the file was uploaded
const dest = path.resolve(UPLOAD_DIR, name)
if(! fse.existsSync(dest)) { fse.moveSync(file.path, dest) ctx.body = {url: `/upload/${name}`.message: 'File uploaded successfully'}}else {
ctx.body = { message: "File already exists"}}})module.exports = router
Copy the codeThe logic is simple: get the name and file fields in the request via koa-body, and copy the file from the temporary directory to the uploaded directory.
3. Implement drag and drop and progress bar
Add a drop event to the div to get information about the file being dragged
mounted() {
this.bindEvent();
},
methods: {
bindEvent() {
const dragEle = this.$refs.drag;
dragEle.addEventListener("dragover".(e) = > {
drag.style.borderColor = "red";
e.preventDefault();
});
dragEle.addEventListener("dragleave".(e) = > {
drag.style.borderColor = "#eee";
e.preventDefault();
});
dragEle.addEventListener("drop".(e) = > {
const fileList = e.dataTransfer.files;
drag.style.borderColor = "#eee";
this.file = fileList[0];
console.log(this.file); e.preventDefault(); }); }},Copy the codeFor the progress bar, simply configure the onUploadProgress option in AXIOS.
const ret = await this.$http.post("/uploadfile", formData, {
onUploadProgress: (progress) = > {
console.log(progress);
this.uploadProgress = ((progress.loaded / progress.total) * 100) | 0; }});Copy the code4. Confirm the binary file format
Whether a file is a picture can be determined by its binary information
Header information:
- gif
GIF89a '47 49 46 38 39 61' GIF87a '47 49 46 38 37 61' - jpeg
FF D8After twoFF D9 - PNG before eight
89 50 4E 47 0D 0A 1A 0A
// upload.vue
async blobToStr(blob) {
return new Promise((resolve) = > {
const fileReader = new FileReader();
fileReader.onload = () = > {
const str = fileReader.result
.split("")
.map((v) = > v.charCodeAt()) // Convert to character encoding
.map((v) = > v.toString(16).toUpperCase()) // Convert to hexadecimal and uppercase
.map((v) = > (v.length < 2 ? "0" + v : v)) // add 0 before less than two digits
.join("");
resolve(str);
};
fileReader.readAsBinaryString(blob);
});
},
async isGif(file) {
// Determine the first six bits of the file
const ret = await this.blobToStr(file.slice(0.6));
console.log(ret);
return ret === "47 49 46 38 39 61" || ret === "47 49 46 38 37 61";
},
async isPng(file) {
'89 50 4E 47 0D 0A 1A 0A'
const ret = await this.blobToStr(file.slice(0.8));
return ret === "89 50 4E 47 0D 0A 1A 0A";
},
async isJpg(file) {
// 'FF D9'
const head = await this.blobToStr(file.slice(0.2));
const tail = await this.blobToStr(file.slice(-2));
return head === "FF D8" && tail === "FF D9";
},
async isImage(file) {
return((await this.isGif(file)) ||
(await this.isPng(file)) ||
(await this.isJpg(file))
);
},
async uploadFile() {
if (!this.file) {
return;
}
// Determine the image format
if(! (await this.isImage(this.file))) {
this.$message.warning("Please select a picture");
return;
}
const formData = new FormData();
formData.append("file".this.file);
formData.append("name".this.file.name);
// Initiate a request
const ret = await this.$http.post("/uploadfile", formData, {
onUploadProgress: (progress) = > {
this.uploadProgress = ((progress.loaded / progress.total) * 100) | 0; }});console.log(ret);
},
Copy the codeThe logic of blobToStr is primarily to read a file as a string through FileReader, and then convert the string to ASCII encoding and then to hexadecimal.
5. Three methods are used to calculate md5
Cut the document into pieces first
async createFileChunks(file) {
const chunks = [];
let cur = 0;
while (cur < file.size) {
chunks.push({ index: cur, chunk: file.slice(cur, cur + CHUNK_SIZE) });
cur += CHUNK_SIZE;
}
return chunks;
},
Copy the codeThe installation relies on Spark-MD5, which appends files and then computs MD5
5.1, web worker
Copy node_modules/spark-md5/spark-md5.min.js to the static directory
Create a hash.js file at static
/ / into the spark
self.importScripts("./spark-md5.min.js");
self.onmessage = e= > {
const { chunks } = e.data;
const spark = new SparkMD5.ArrayBuffer();
let len = chunks.length;
const loadNext = cur= > {
const fileReader = new FileReader();
fileReader.readAsArrayBuffer(chunks[cur].chunk);
fileReader.onload = () = > {
spark.append(fileReader.result);
cur++;
if (cur < len) {
const progress = Number(((cur * 100) / chunks.length).toFixed(2));
self.postMessage({ progress });
loadNext(cur);
} else {
self.postMessage({ progress: 100.hash: spark.end() }); }}; }; loadNext(0);
};
Copy the codeCreate a method calculateHashWorker
async calculateHashWorker(chunks) {
return new Promise(resolve= > {
const worker = new Worker("hash.js");
worker.postMessage({ chunks });
worker.onmessage = e= > {
const { progress, hash } = e.data;
if(! hash) {this.hashProgress = progress;
} else {
this.hashProgress = 100; resolve(hash); }}; }); }Copy the codeIdeas: A worker is created first, and then sends chunks to the worker via postMessage. After receiving the data, the worker processes only one chunk at a time through the loadNext method. After processing the data, the worker sends the progress back to the main process until all chunks are processed and the hash is returned.
5.2, requestIdleCallback
Spark-md5 is used to perform MD5 calculation on the chunks
async calculateHashIdle(chunks) {
return new Promise(resolve= > {
let count = 0;
const spark = new sparkMD5.ArrayBuffer();
const appendToSpark = file= > {
return new Promise(resolve= > {
const fileReader = new FileReader();
fileReader.onload = () = > {
spark.append(fileReader.result);
resolve();
};
fileReader.readAsArrayBuffer(file);
});
};
const workLoop = async deadling => {
while (count < chunks.length && deadling.timeRemaining() > 1) {
// Add Chunk to Spark
await appendToSpark(chunks[count].chunk);
count++;
if (count < chunks.length) {
this.hashProgress = Number(
((count * 100) / chunks.length).toFixed(2)); }else {
this.hashProgress = 100; resolve(spark.end()); }}window.requestIdleCallback(workLoop);
};
window.requestIdleCallback(workLoop);
});
},
Copy the code5.3. Sampling hash calculation
async calculateHashSample(chunks) {
return new Promise(resolve= > {
let tempChunks = [];
/ / interception chunks
for (let i = 0; i < chunks.length; i++) {
let chunk = chunks[i].chunk;
if (i === 0 || i === chunks.length - 1) {
tempChunks.push(chunk);
} else {
const mid = CHUNK_SIZE >> 1;
tempChunks.push(chunk.slice(0.2));
tempChunks.push(chunk.slice(mid, mid + 2));
tempChunks.push(chunk.slice(-2)); }}// Add blob to Spark via fileReader
const fileReader = new FileReader();
const spark = new sparkMD5.ArrayBuffer();
fileReader.readAsArrayBuffer(new Blob(tempChunks));
fileReader.onload = () = > {
spark.append(fileReader.result);
resolve(spark.end());
};
});
},
Copy the codeWhen slicing, only the first and last blocks are cut, and only the first, middle and last two bytes are taken from the middle block
6. Upload and merge slices
The front-end implementation
// upload.vue <template> ... <div class="cube-container" :style="{ width: cubeWidth + 'px' }"> <div class="cube" v-for="chunk in chunks" :key="chunk.name"> <div :class="{ uploading: chunk.progress > 0 && chunk.progress < 100, success: chunk.progress === 100, error: chunk.progress === -1, }" :style="{ height: chunk.progress + '%' }" > <i class="el-icon-loading" style="color: #f56c6c" v-if="chunk.progress < 100 && chunk.progress > 0" ></i> </div> </div> </div> </template> <script> export default { data() { return { uploadProgress: 0, chunks: [], }; }, mounted() { this.bindEvent(); }, computed: { cubeWidth() { return Math.ceil(Math.sqrt(this.chunks.length)) * 16; }, ext() { return this.file.name.split(".")[1]; }, }, methods:{ ... async uploadFile() { if (! this.file) { return; } // Determine the image format if (! (await this.isimage (this.file))) {this.$message. Warning (" Please select image "); return; } // slice const chunks = await this.createFilechunks (this.file); // Calculate hash const hash = await this.calculateHashWorker(chunks); // const hash2 = await this.calculateHashIdle(chunks); // const hash3 = await this.calculateHashSample(chunks); this.hash = hash; this.chunks = chunks.map((chunk, index) => { const name = `${hash}_${index}`; return { name, hash, index, chunk: chunk.chunk, progress: 0, }; }); await this.uploadChunks(); await this.mergeRequest(); }, async uploadChunks() { return new Promise((resolve) => { const chunks = this.chunks; Const requests = chunk.map ((chunk) => {// build formData const formData = new formData (); formData.append("chunk", chunk.chunk); formData.append("name", chunk.name); formData.append("hash", chunk.hash); return { formData, chunk }; Return this.$http.post("/uploadChunk", formData, {onUploadProgress: (progressEvent) => { let complete = ((progressEvent.loaded / progressEvent.total) * 100) | 0; chunk.progress = complete; }}); }); Promise.all(requests).then(() => { resolve(); }); }); }, async mergeRequest() { await this.$http.post("/mergefile", { hash: this.hash, ext: this.ext, size: CHUNK_SIZE, }); } </script> <style > ... .cube-container { .cube { width: 14px; height: 14px; line-height: 12px; border: 1px black solid; background: #eee; float: left; >.success { background: green; } >.uploading { background: blue; } >.error { background: red; } } } </style>Copy the codeAdd the block uploading progress to template and add the uploadChunks method for uploading chunks in batches. After uploading all chunks successfully, merge all chunks through the mergeRequest method.
The backend implementation
The backend needs to provide two methods, one for uploading the chunks and one for merging the chunks
// server/upload.js. router.post("/uploadChunk".async ctx => {
const { hash, name } = ctx.request.body;
const file = ctx.request.files.chunk;
const uploadDir = path.resolve(UPLOAD_DIR, hash);
if(! fse.existsSync(uploadDir)) { fse.mkdirSync(uploadDir); } fse.moveSync(file.path, path.resolve(uploadDir, name)); ctx.body = {code: 0.message: "Upload successful" };
});
router.post("/mergefile".async ctx => {
const { hash, ext } = ctx.request.body;
const filename = path.resolve(UPLOAD_DIR, `${hash}.${ext}`);
const dirname = path.resolve(UPLOAD_DIR, hash);
if(! fse.existsSync(dirname)) { ctx.body = {code: 0.message: "File directory does not exist" };
return;
}
const uploadList = fse
.readdirSync(dirname)
.map(v= > path.resolve(dirname, v))
.sort((a, b) = > a.split("_") [1] - b.split("_") [1]);
for (let i = 0; i < uploadList.length; i++) {
fse.appendFileSync(filename, fse.readFileSync(uploadList[i]));
fse.unlink(uploadList[i]);
}
ctx.body = { code: 0.message: "Merger successful" };
});
Copy the code7. Second transmission and resumable transmission
The implementation logic of second transmission: The front end sends a request to the back end to ask whether the file exists. If the file exists, a message is displayed indicating that the second transmission is successful.
The implementation logic of resumable upload: the front-end sends a request to the back-end to ask whether the file exists or has been uploaded. The back-end returns the uploaded fragment. The front-end filters out the uploaded fragment before uploading.
Checkfile and return the event and uploadList.
//server/upload.js. router.get("/checkfile".async ctx => {
const { hash, ext } = ctx.request.query;
const filePath = path.resolve(UPLOAD_DIR, `${hash}.${ext}`);
let uploaded = false;
let uploadList = [];
if (fse.existsSync(filePath)) {
uploaded = true;
} else {
// Read the directory
uploadList = getUploadList(path.resolve(UPLOAD_DIR, hash));
}
ctx.body = { uploaded, uploadList };
});
function getUploadList(dirPath) {
return fse.existsSync(dirPath) ? fse.readdirSync(dirPath) : [];
}
Copy the codeThe front end
// upload.vue
async uploadFile(){.../ / section
const chunks = await this.createFileChunks(this.file);
/ / calculate the hash
const hash = await this.calculateHashWorker(chunks);
// const hash2 = await this.calculateHashIdle(chunks);
// const hash3 = await this.calculateHashSample(chunks);
// Check whether the file has been uploaded
const { uploaded, uploadList } = await this.checkFile();
if (uploaded) {
this.$message.success("Transmission succeeded in seconds");
return; }...await this.uploadChunks(uploadList);
await this.mergeRequest();
},
async checkFile() {
const ret = await this.$http.get(
`/checkfile? hash=The ${this.hash}&ext=The ${this.ext}`
);
console.log(ret);
return ret.data;
},
async uploadChunks(uploadList) {
return new Promise((resolve) = > {
const chunks = this.chunks;
const requests = chunks
// Set the progress of uploaded chunks to 100
.map((chunk) = > {
if (uploadList.includes(chunk.name)) {
chunk.progress = 100;
}
return chunk;
})
// Filter out chunks that have been uploaded
.filter((chunk) = >chunk.progress ! = =100)
.map((chunk) = > {
// Build the form data
const formData = new FormData();
formData.append("chunk", chunk.chunk);
formData.append("name", chunk.name);
formData.append("hash", chunk.hash);
return{ formData, chunk }; })... }Copy the code8. Concurrency control and error retry
async uploadChunks(uploadList) {
const chunks = this.chunks;
// return new Promise(resolve => {
const requests = chunks
.map((chunk) = > {
if (uploadList.includes(chunk.name)) {
chunk.progress = 100;
}
return chunk;
})
.filter((chunk) = >chunk.progress ! = =100)
.map((chunk) = > {
const formData = new FormData();
formData.append("chunk", chunk.chunk);
formData.append("name", chunk.name);
formData.append("hash", chunk.hash);
return { formData, chunk };
});
await this.sendRequests(requests);
},
async sendRequests(tasks, limit = 4) {
return new Promise((resolve, reject) = > {
let isStop = false;
const len = tasks.length;
let count = 0;
const next = async() = > {if (isStop) {
return;
}
const task = tasks.shift();
if (task) {
task.error = task.error || 0;
const { chunk, formData } = task;
try {
await this.$http.post("/uploadChunk", formData, {
onUploadProgress: (progressEvent) = > {
let complete =
((progressEvent.loaded / progressEvent.total) * 100) | 0; chunk.progress = complete; }}); count++;if (count < len) {
next();
} else{ resolve(); }}catch (e) {
// Display error
chunk.progress = -1;
if (task.error < 3) {
task.error++;
tasks.unshift(task);
next();
} else {
isStop = true; reject(); }}}};while (limit > 0) { next(); limit--; }}); },Copy the codeConcurrency control principle: through the while loop to control the number of simultaneous requests, when the request comes back and then the next request.
Error retry Principle: When an error occurs in a request, the number of errors in the request is increased by one, and the task is added to the task queue. If more than three errors occur, isStop is set to true to end all requests.
Making the address
Finally, the github address is attached. Interested students can download and use it or study it. If there is any problem or bad improvement in the demo, they can also discuss with each other. If there is a harvest, welcome to start, you can also leave a message feedback at any time