First, write first
I believe that all of you have encountered the “large file upload” scenario in your actual business. In this scenario, we can’t just dump large files directly to the server for processing, which will have a huge impact on the server’s performance and upload speed is too slow. Therefore, we will adopt the technical solution of “large file fragment upload” to upload files as quickly as possible and have as little impact on server performance as possible.
Just recently while taking the advantage of spare time, a detailed understanding of the technical details of “large file sharding upload”, found that some of the use experience of the sharding upload library are not very good, so here from scratch handwritten a large file sharding upload library, one is to deepen understanding, the second is convenient for everyone to use directly afterwards.
Two, large file fragmentation upload technical scheme
Generally speaking, uploading large file fragments mainly involves the following steps:
1. Front-end calculation file MD5. The MD5 calculation is used to check whether the files uploaded to the server are the same as those uploaded by the user. You can also perform operations such as second transmission based on MD5.
2. The front-end sends the initialization request, and the back-end initializes the upload. After the MD5 of the file is calculated, you can enter the initial upload step. In this step, the front end initiates the initialization request, including the MD5 and file name of the file, and the back end initializes the directory of the fragment file according to the MD5.
3. The front-end divides files and transmits fragments to the back-end. Needless to say, the front end will divide the file into multiple pieces and upload the file according to certain policies. If a fragment fails to be uploaded, the file needs to be uploaded again.
4. The front-end sends the end upload request, and the back-end merges fragments. After all file fragments are successfully sent, the front-end sends an end upload request. After receiving the request, the back-end merges the existing file fragments to generate a file, and checks whether the MD5 of the generated file is the same as that of the initialized md5.
It is worth noting that when a file is large, it consumes a lot of memory to calculate MD5, fragment and merge files according to the file directly (or even eat up the memory directly). Therefore, in these three steps, you need to use streams to reduce the memory consumption.
Third, easy – file – uploader
The easy-file-uploader source code address is Typescript’s “out-of-the-box large file sharding upload library”
You can directly click the above address to view readme.md.
So without further ado, let’s see how I implemented this library.
Easy -file- Uploader -server implementation process
From the “large file sharding upload technology solution”, we can make it clear that the back end should provide the following basic capabilities:
1. Initialize file upload
2. Receive file fragments
3. Merge file fragments
Second, in order to use the experience, we need to provide the following additional capabilities:
4. Obtain the uploaded fragment information
5. Clear the shard directory (for canceling uploads)
So we’ll start by writing a FileUploaderServer class that provides these capabilities. Thus, when using easy-file-uploader-Server, developers simply need to instantiate the FileUploaderServer class and use the methods provided by this class in the interface.
This is done to provide better extensibility — after all, developers might implement interfaces with express/ KOA/native NodeJS frameworks, and if we implement them one by one… It’s too bad for maintenance.
So we can quickly write the big frame of the class, which looks something like this:
interface IFileUploaderOptions {
tempFileLocation: string; // Fragment storage path
mergedFileLocation: string; // The merged file path
}
class FileUploaderServer {
private fileUploaderOptions: IFileUploaderOptions;
/** * Initializes the file fragment upload, actually calculates an MD5 based on fileName and time, and creates a new folder *@param FileName fileName *@returns Upload Id * /
public async initFilePartUpload(fileName: string) :Promise<string> {}
/ * * * upload shard, is actually writing partFile uploadId corresponding folder, written to the file naming format for ` partIndex | md5 ` *@param UploadId uploadId *@param PartIndex Indicates the fragment number *@param PartFile Fragment content *@returns Shard md5 * /
public async uploadPartFile(
uploadId: string.partIndex: number.partFile: Buffer,
): Promise<string> {}
/** * to retrieve the uploaded shard information, actually read the contents of the folder *@param UploadId uploadId *@returns Uploaded fragment information */
public async listUploadedPartFile(
uploadId: string,
): Promise<IUploadPartInfo[]> {}
/** * Cancel file upload, hard delete will directly delete folder, soft delete will change folder name *@param UploadId uploadId *@param DeleteFolder Specifies whether to directly delete the folder */
async cancelFilePartUpload(
uploadId: string.deleteFolder: boolean = false,
): Promise<void> {}
/** * All fragments are read together, checked by MD5, and saved to a new path. *@param UploadId uploadId *@param FileName fileName *@param Md5 File MD5 *@returns File storage path */
async finishFilePartUpload(
uploadId: string.fileName: string.md5: string,
): Promise<IMergedFileInfo> {}
}
Copy the code4-1. Initialize file upload
During initial upload, we create a directory under the tempFileLocation directory (sharding directory) based on MD5 to store the uploaded shard. ${fileName}-${date.now ()} -${date.now ()}
/** * Initializes the file fragment upload, actually calculates an MD5 based on fileName and time, and creates a new folder *@param FileName fileName *@returns Upload Id * /
public async initFilePartUpload(fileName: string) :Promise<string> {
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadId = calculateMd5(`${fileName}-The ${Date.now()}`);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if (uploadFolderExist) {
throw new FolderExistException(
'found same upload folder, maybe you meet hash collision',); }await fse.mkdir(uploadFolderPath);
return uploadId;
}
Copy the code4-2. Receiving file fragments
When receiving file fragmentation, we first get a shard storage location, and then calculate the md5 shard, then subdivision named ${partIndex} | ${partFileMd5}. Part, stored in the corresponding directory.
/ * * * upload shard, is actually writing partFile uploadId corresponding folder, written to the file naming format for ` partIndex | md5 ` *@param UploadId uploadId *@param PartIndex Indicates the fragment number *@param PartFile Fragment content *@returns Shard md5 * /
public async uploadPartFile(
uploadId: string.partIndex: number.partFile: Buffer,
): Promise<string> {
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if(! uploadFolderExist) {throw new NotFoundException('not found upload folder');
}
const partFileMd5 = calculateMd5(partFile);
const partFileLocation = path.join(
uploadFolderPath,
`${partIndex}|${partFileMd5}.part`,);await fse.writeFile(partFileLocation, partFile);
return partFileMd5;
}
Copy the code4-3. Merge file fragments
The most important method to merge shards is the mergePartFile method, which uses readStream and writeStream to read/write shards. This has the advantage of minimizing memory footprint. Also, use the pipe method provided by MultiStream to ensure the order of the streams.
export async function mergePartFile(
files: IFileInfo[],
mergedFilePath: string.) :Promise<void> {
const fileList = files.map((item) = > {
const [index] = item.name.replace(/\.part$/.' ').split('|');
return {
index: parseInt(index),
path: item.path,
};
});
const sortedFileList = fileList.sort((a, b) = > {
return a.index - b.index;
});
const sortedFilePathList = sortedFileList.map((item) = > item.path);
merge(sortedFilePathList, mergedFilePath);
}
function merge(inputPathList: string[], outputPath: string) {
const fd = fse.openSync(outputPath, 'w+');
const writeStream = fse.createWriteStream(outputPath);
const readStreamList = inputPathList.map((path) = > {
return fse.createReadStream(path);
});
return new Promise((resolve, reject) = > {
const multiStream = new MultiStream(readStreamList);
multiStream.pipe(writeStream);
multiStream.on('end'.() = > {
fse.closeSync(fd);
resolve(true);
});
multiStream.on('error'.() = > {
fse.closeSync(fd);
reject(false);
});
});
}
Copy the codeSo with mergePartFile method, merge file fragment finishFilePartUpload method is also ready to come out, on the basis of mergePartFile, add file save path acquisition and MD5 verification.
/** * All fragments are read together, checked by MD5, and saved to a new path. *@param UploadId uploadId *@param FileName fileName *@param Md5 File MD5 *@returns File storage path */
async finishFilePartUpload(
uploadId: string.fileName: string.md5: string,
): Promise<IMergedFileInfo> {
const { mergedFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(mergedFileLocation);
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if(! uploadFolderExist) {throw new NotFoundException('not found upload folder');
}
const dirList = await listDir(uploadFolderPath);
const files = dirList.filter((item) = > item.path.endsWith('.part'));
const mergedFileDirLocation = path.join(mergedFileLocation, md5);
await fse.ensureDir(mergedFileDirLocation);
const mergedFilePath = path.join(mergedFileDirLocation, fileName);
await mergePartFile(files, mergedFilePath);
await wait(1000); // Wait a while, otherwise empty files will be read during md5 calculation
const mergedFileMd5 = await calculateFileMd5(mergedFilePath);
if(mergedFileMd5 ! == md5) {throw new Md5Exception('md5 checked failed');
}
return {
path: mergedFilePath,
md5,
};
}
Copy the code4-4. Obtain the uploaded fragment information
To obtain the shard information that has been uploaded is actually to read all the shard files with suffix part in uploadId directory.
/** * to retrieve the uploaded shard information, actually read the contents of the folder *@param UploadId uploadId *@returns Uploaded fragment information */
public async listUploadedPartFile(
uploadId: string,
): Promise<IUploadPartInfo[]> {
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if(! uploadFolderExist) {throw new NotFoundException('not found upload folder');
}
const dirList = await listDir(uploadFolderPath);
const uploadPartInfo = dirList.map((item: IFileInfo) = > {
const [index, md5] = item.name.replace(/\.part$/.' ').split('|');
return {
path: item.path,
index: parseInt(index),
md5,
};
});
return uploadPartInfo;
}
function listDir(path: string) :Promise<IFileInfo[] >{
const items = await fse.readdir(path);
return Promise.all(
items
.filter((item: string) = >! item.startsWith('. '))
.map(async (item: string) = > {return {
name: item,
path: `${path}/${item}`}; })); }Copy the code4-5. Clear the fragment storage directory
Cleaning the shard directory is actually very simple, if soft delete, just change the directory name. If it is a hard delete, then delete the directory.
/** * Cancel file upload, hard delete will directly delete folder, soft delete will change folder name *@param UploadId uploadId *@param DeleteFolder Specifies whether to directly delete the folder */
async cancelFilePartUpload(
uploadId: string.deleteFolder: boolean = false,
): Promise<void> {
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if(! uploadFolderExist) {throw new NotFoundException('not found upload folder');
}
if (deleteFolder) {
await fse.remove(uploadFolderPath);
} else {
await fse.rename(uploadFolderPath, `${uploadFolderPath}[removed]`); }}Copy the code4-6. Detailed code
Putting the above code together, the FileUploaderServer class is complete. More detailed code can be viewed at github above: click here
import * as path from 'path';
import * as fse from 'fs-extra';
import {
calculateFileMd5,
calculateMd5,
IFileInfo,
listDir,
mergePartFile,
wait,
} from './util';
import {
FolderExistException,
Md5Exception,
NotFoundException,
} from './exception';
const DEAFULT_TEMP_FILE_LOCATION = path.join(__dirname, './upload_file');
const DEAFULT_MERGED_FILE_LOCATION = path.join(__dirname, './merged_file');
const DEFAULT_OPTIONS = {
tempFileLocation: DEAFULT_TEMP_FILE_LOCATION,
mergedFileLocation: DEAFULT_MERGED_FILE_LOCATION,
};
export interface IFileUploaderOptions {
tempFileLocation: string;
mergedFileLocation: string;
}
export interface IUploadPartInfo {
path: string;
index: number;
md5: string;
}
export interface IMergedFileInfo {
path: string;
md5: string;
}
export class FileUploaderServer {
private fileUploaderOptions: IFileUploaderOptions;
constructor(options: IFileUploaderOptions) {
this.fileUploaderOptions = Object.assign(DEFAULT_OPTIONS, options);
}
public getOptions() {
return this.fileUploaderOptions;
}
/** * Initializes the file fragment upload, actually calculates an MD5 based on fileName and time, and creates a new folder *@param FileName fileName *@returns Upload Id * /
public async initFilePartUpload(fileName: string) :Promise<string> {
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadId = calculateMd5(`${fileName}-The ${Date.now()}`);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if (uploadFolderExist) {
throw new FolderExistException(
'found same upload folder, maybe you meet hash collision',); }await fse.mkdir(uploadFolderPath);
return uploadId;
}
/ * * * upload shard, is actually writing partFile uploadId corresponding folder, written to the file naming format for ` partIndex | md5 ` *@param UploadId uploadId *@param PartIndex Indicates the fragment number *@param PartFile Fragment content *@returns Shard md5 * /
public async uploadPartFile(
uploadId: string.partIndex: number.partFile: Buffer,
): Promise<string> {
const uploadFolderPath = await this.getUploadFolder(uploadId);
const partFileMd5 = calculateMd5(partFile);
const partFileLocation = path.join(
uploadFolderPath,
`${partIndex}|${partFileMd5}.part`,);await fse.writeFile(partFileLocation, partFile);
return partFileMd5;
}
/** * to retrieve the uploaded shard information, actually read the contents of the folder *@param UploadId uploadId *@returns Uploaded fragment information */
public async listUploadedPartFile(
uploadId: string,
): Promise<IUploadPartInfo[]> {
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if(! uploadFolderExist) {throw new NotFoundException('not found upload folder');
}
const dirList = await listDir(uploadFolderPath);
const uploadPartInfo = dirList.map((item: IFileInfo) = > {
const [index, md5] = item.name.replace(/\.part$/.' ').split('|');
return {
path: item.path,
index: parseInt(index),
md5,
};
});
return uploadPartInfo;
}
/** * Cancel file upload, hard delete will directly delete folder, soft delete will change folder name *@param UploadId uploadId *@param DeleteFolder Specifies whether to directly delete the folder */
async cancelFilePartUpload(
uploadId: string.deleteFolder: boolean = false,
): Promise<void> {
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if(! uploadFolderExist) {throw new NotFoundException('not found upload folder');
}
if (deleteFolder) {
await fse.remove(uploadFolderPath);
} else {
await fse.rename(uploadFolderPath, `${uploadFolderPath}[removed]`); }}/** * All fragments are read together, checked by MD5, and saved to a new path. *@param UploadId uploadId *@param FileName fileName *@param Md5 File MD5 *@returns File storage path */
async finishFilePartUpload(
uploadId: string.fileName: string.md5: string,
): Promise<IMergedFileInfo> {
const { mergedFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(mergedFileLocation);
const uploadFolderPath = await this.getUploadFolder(uploadId);
const dirList = await listDir(uploadFolderPath);
const files = dirList.filter((item) = > item.path.endsWith('.part'));
const mergedFileDirLocation = path.join(mergedFileLocation, md5);
await fse.ensureDir(mergedFileDirLocation);
const mergedFilePath = path.join(mergedFileDirLocation, fileName);
await mergePartFile(files, mergedFilePath);
await wait(1000); // Wait a while, otherwise empty files will be read during md5 calculation
const mergedFileMd5 = await calculateFileMd5(mergedFilePath);
if(mergedFileMd5 ! == md5) {throw new Md5Exception('md5 checked failed');
}
return {
path: mergedFilePath,
md5,
};
}
/** * Get the path to upload the folder *@param UploadId uploadId *@returns Folder path */
private async getUploadFolder(uploadId: string) :Promise<string> {
const { tempFileLocation } = this.fileUploaderOptions;
await fse.ensureDir(tempFileLocation);
const uploadFolderPath = path.join(tempFileLocation, uploadId);
const uploadFolderExist = fse.existsSync(uploadFolderPath);
if(! uploadFolderExist) {throw new NotFoundException('not found upload folder');
}
returnuploadFolderPath; }}Copy the codeFive, easy file-uploader-client implementation process
Once the back-end logic is written, we can start writing the front-end logic.
As mentioned above, in order to meet the developers’ needs for scalability, easy-file-uploader-server provides “large file shard upload” capability instead of directly providing “large file shard upload” interface. As a result, when designing easy-file-uploader-client, you cannot make requests directly. Therefore, easy-file-Uploader-Client is only expected to provide control over the sharding upload process at the beginning of its design, and will not implement the specific upload function.
Easy-file-uploader – Client provides the following basic capabilities:
1. Md5 calculation and sharding of files 2. Support user-defined upload functions and control the execution process of these upload functions.
Therefore, we will first write a FileUploaderClient class that provides these capabilities. In this way, when using easy-file-uploader-Client, developers simply need to instantiate the FileUploaderClient class and use the provided capabilities when uploading. Of course, if the user wants to control the execution flow of the upload function himself, then he can only use the “file MD5 calculation and sharding” capability.
5-1. Md5 calculation and fragmentation of files
Implement the MD5 calculation function of files Spark-MD5 is used to calculate the MD5 value of files. For sharding, use the browser’s built-in FileReader to read the file and then use the browser’s built-in APIblobSlice to slice the file.
It is worth noting that when the file size is large, “directly calculate MD5 for the whole file” and “directly load the whole file and shard” are both performance-intensive operations that take a long time. In this case, we also need to read the file through the input stream like easy-file-uploader-server.
/** * Shards the file object and calculates md5 * based on the shards@param File File to be uploaded *@returns Return md5 and shard list */
public async getChunkListAndFileMd5(
file: File,
): PromiseThe < {md5: string; chunkList: Blob[] }> {
return new Promise((resolve, reject) = > {
let currentChunk = 0;
const chunkSize = this.fileUploaderClientOptions.chunkSize;
const chunks = Math.ceil(file.size / chunkSize);
const spark = new SparkMD5.ArrayBuffer();
const fileReader = new FileReader();
const blobSlice = getBlobSlice();
const chunkList: Blob[] = [];
fileReader.onload = function (e) {
if(e? .target? .resultinstanceof ArrayBuffer) {
spark.append(e.target.result);
}
currentChunk++;
if (currentChunk < chunks) {
loadNextChunk();
} else {
const computedHash = spark.end();
resolve({ md5: computedHash, chunkList }); }}; fileReader.onerror =function (e) {
console.warn('read file error', e);
reject(e);
};
function loadNextChunk() {
const start = currentChunk * chunkSize;
const end =
start + chunkSize >= file.size ? file.size : start + chunkSize;
const chunk = blobSlice.call(file, start, end);
chunkList.push(chunk);
fileReader.readAsArrayBuffer(chunk);
}
loadNextChunk();
});
}
Copy the code5-2. Control the upload process
In fact, the upload process is relatively simple. First of all, we need developers to implement initFilePartUploadFunc, uploadPartFileFunc, finishFilePartUploadFunc three functions. They are then passed to the FileUploaderClient as configuration items. Finally, we provide a function uploadFile, and execute the three functions in the configuration item in sequence to complete the whole process of uploading large file fragments.
The overall uploading process is simple: 1. Run getChunkListAndFileMd5 to fragment the file and calculate MD5. 2. Run initFilePartUploadFunc to initialize file uploading. 3. Run uploadPartFileFunc once for each shard. If this fails, add it to retryList. 4. Retry the fragments that fail to upload the retryList. 5. Run finishFilePartUploadFunc to upload the file.
/** * How to upload a file, * can only be used if the FileUploaderClient configuration item is passed in requestOptions GetChunkListAndFileMd5, initFilePartUploadFunc in the configuration item, uploadPartFileFunc in the configuration item, and finishFilePartUploadFunc * in the configuration item are executed in sequence After the execution is complete, the system returns the upload result. If fragment uploading fails, the system automatically retries *@param File File to be uploaded *@returns FinishFilePartUploadFunc function Promise resolve value */
public async uploadFile(file: File): Promise<any> {
const requestOptions = this.fileUploaderClientOptions.requestOptions;
const { md5, chunkList } = await this.getChunkListAndFileMd5(file);
const retryList = [];
if( requestOptions? .retryTimes ===undefined| |! requestOptions? .initFilePartUploadFunc || ! requestOptions? .uploadPartFileFunc || ! requestOptions? .finishFilePartUploadFunc ) {throw Error(
'invalid request options, need retryTimes, initFilePartUploadFunc, uploadPartFileFunc and finishFilePartUploadFunc',); }await requestOptions.initFilePartUploadFunc();
for (let index = 0; index < chunkList.length; index++) {
try {
await requestOptions.uploadPartFileFunc(chunkList[index], index);
} catch (e) {
console.warn(`${index} part upload failed`); retryList.push(index); }}for (let retry = 0; retry < requestOptions.retryTimes; retry++) {
if (retryList.length > 0) {
console.log(`retry start, times: ${retry}`);
for (let a = 0; a < retryList.length; a++) {
const blobIndex = retryList[a];
try {
await requestOptions.uploadPartFileFunc(
chunkList[blobIndex],
blobIndex,
);
retryList.splice(a, 1);
} catch (e) {
console.warn(
`${blobIndex} part retry upload failed, times: ${retry}`,); }}}}if (retryList.length === 0) {
return await requestOptions.finishFilePartUploadFunc(md5);
} else {
throw Error(
`upload failed, some chunks upload failed: The ${JSON.stringify(
retryList,
)}`,); }}Copy the code5-3. General code
Putting the above code together, the FileUploaderClient class is complete. More detailed code can be viewed at github above: click here
import SparkMD5 from 'spark-md5';
import { getBlobSlice } from './util';
const DEFAULT_CHUNK_SIZE = 5 * 1024 * 1024;
const DEFAULT_OPTIONS = {
chunkSize: DEFAULT_CHUNK_SIZE,
};
export interface IFileUploaderClientOptions {
chunkSize: number; requestOptions? : {retryTimes: number;
initFilePartUploadFunc: () = > Promise<any>;
uploadPartFileFunc: (chunk: Blob, index: number) = > Promise<any>;
finishFilePartUploadFunc: (md5: string) = > Promise<any>;
};
}
export class FileUploaderClient {
fileUploaderClientOptions: IFileUploaderClientOptions;
constructor(options: IFileUploaderClientOptions) {
this.fileUploaderClientOptions = Object.assign(DEFAULT_OPTIONS, options);
}
/** * Shards the file object and calculates md5 * based on the shards@param File File to be uploaded *@returns Return md5 and shard list */
public async getChunkListAndFileMd5(
file: File,
): PromiseThe < {md5: string; chunkList: Blob[] }> {
return new Promise((resolve, reject) = > {
let currentChunk = 0;
const chunkSize = this.fileUploaderClientOptions.chunkSize;
const chunks = Math.ceil(file.size / chunkSize);
const spark = new SparkMD5.ArrayBuffer();
const fileReader = new FileReader();
const blobSlice = getBlobSlice();
const chunkList: Blob[] = [];
fileReader.onload = function (e) {
if(e? .target? .resultinstanceof ArrayBuffer) {
spark.append(e.target.result);
}
currentChunk++;
if (currentChunk < chunks) {
loadNextChunk();
} else {
const computedHash = spark.end();
resolve({ md5: computedHash, chunkList }); }}; fileReader.onerror =function (e) {
console.warn('read file error', e);
reject(e);
};
function loadNextChunk() {
const start = currentChunk * chunkSize;
const end =
start + chunkSize >= file.size ? file.size : start + chunkSize;
const chunk = blobSlice.call(file, start, end);
chunkList.push(chunk);
fileReader.readAsArrayBuffer(chunk);
}
loadNextChunk();
});
}
/** * How to upload a file, * can only be used if the FileUploaderClient configuration item is passed in requestOptions GetChunkListAndFileMd5, initFilePartUploadFunc in the configuration item, uploadPartFileFunc in the configuration item, and finishFilePartUploadFunc * in the configuration item are executed in sequence After the execution is complete, the system returns the upload result. If fragment uploading fails, the system automatically retries *@param File File to be uploaded *@returns FinishFilePartUploadFunc function Promise resolve value */
public async uploadFile(file: File): Promise<any> {
const requestOptions = this.fileUploaderClientOptions.requestOptions;
const { md5, chunkList } = await this.getChunkListAndFileMd5(file);
const retryList = [];
if( requestOptions? .retryTimes ===undefined| |! requestOptions? .initFilePartUploadFunc || ! requestOptions? .uploadPartFileFunc || ! requestOptions? .finishFilePartUploadFunc ) {throw Error(
'invalid request options, need retryTimes, initFilePartUploadFunc, uploadPartFileFunc and finishFilePartUploadFunc',); }await requestOptions.initFilePartUploadFunc();
for (let index = 0; index < chunkList.length; index++) {
try {
await requestOptions.uploadPartFileFunc(chunkList[index], index);
} catch (e) {
console.warn(`${index} part upload failed`); retryList.push(index); }}for (let retry = 0; retry < requestOptions.retryTimes; retry++) {
if (retryList.length > 0) {
console.log(`retry start, times: ${retry}`);
for (let a = 0; a < retryList.length; a++) {
const blobIndex = retryList[a];
try {
await requestOptions.uploadPartFileFunc(
chunkList[blobIndex],
blobIndex,
);
retryList.splice(a, 1);
} catch (e) {
console.warn(
`${blobIndex} part retry upload failed, times: ${retry}`,); }}}}if (retryList.length === 0) {
return await requestOptions.finishFilePartUploadFunc(md5);
} else {
throw Error(
`upload failed, some chunks upload failed: The ${JSON.stringify(
retryList,
)}`,); }}}Copy the codeSix, try it out!
6-1, the server side
Easy -file-uploader-server router configuration
const Koa = require('koa')
const bodyParser = require('koa-bodyparser')
const router = require('./router')
const cors = require('@koa/cors')
const staticResource = require('koa-static')
const path = require('path')
const KoaRouter = require('koa-router')
const multer = require('@koa/multer')
const path = require('path')
const { FileUploaderServer } = require('easy-file-uploader-server')
const PORT = 10001
const app = new Koa()
const upload = multer()
const router = KoaRouter()
const fileUploader = new FileUploaderServer({
tempFileLocation: path.join(__dirname, './public/tempUploadFile'),
mergedFileLocation: path.join(__dirname, './public/mergedUploadFile'),
})
router.post('/api/initUpload'.async (ctx, next) => {
const { name } = ctx.request.body
const uploadId = await fileUploader.initFilePartUpload(name)
ctx.body = { uploadId }
await next()
})
router.post('/api/uploadPart', upload.single('partFile'), async (ctx, next) => {
const { buffer } = ctx.file
const { uploadId, partIndex } = ctx.request.body
const partFileMd5 = await fileUploader.uploadPartFile(uploadId, partIndex, buffer)
ctx.body = { partFileMd5 }
await next()
})
router.post('/api/finishUpload'.async (ctx, next) => {
const { uploadId, name, md5 } = ctx.request.body
const { path: filePathOnServer } = await fileUploader.finishFilePartUpload(uploadId, name, md5)
const suffix = filePathOnServer.split('/public/') [1]
ctx.body = { path: suffix }
await next()
})
app.use(<span data-word-id="828" class="abbreviate-word">cors</span>())
app.use(bodyParser())
app.use(staticResource(path.join(__dirname, 'public')))
app.use(router.routes())
app.use(router.allowedMethods())
app.listen(PORT)
console.log(`app run in port: ${PORT}`)
console.log(`visit http://localhost:${PORT}/index.html to start demo`)
Copy the code6-2, the client side
Use react to write a client. Focus on the logic in the APP component.
import { useRef, useState } from 'react'
import './App.<span data-word-id="51586836" class="abbreviate-word">css</span>'
import axios from 'axios'
import { FileUploaderClient } from 'easy-file-uploader-client'
const HOST = 'http://localhost:10001/'
function App() {
const fileInput = useRef(null)
const [url, setUrl] = useState<string> (' ')
let uploadId = ' '
const fileUploaderClient = new FileUploaderClient({
chunkSize: 2 * 1024 * 1024.// 2MB
requestOptions: {
retryTimes: 2.initFilePartUploadFunc: async() = > {const fileName = (fileInput.current as any).files[0].name
const { data } = await axios.post(`${HOST}api/initUpload`, {
name: fileName,
})
uploadId = data.uploadId
console.log('Initialization upload completed')
setUrl(' ')},uploadPartFileFunc: async (chunk: Blob, index: number) = > {const formData = new FormData()
formData.append('uploadId', uploadId)
formData.append('partIndex', index.toString())
formData.append('partFile', chunk)
await axios.post(`${HOST}api/uploadPart`, formData, {
headers: { 'Content-Type': 'multipart/form-data'}})console.log('Upload sharding${index}Complete `)},finishFilePartUploadFunc: async (md5: string) = > {const fileName = (fileInput.current as any).files[0].name
const { data } = await axios.post(`${HOST}api/finishUpload`, {
name: fileName,
uploadId,
md5,
})
console.log('Upload complete, storage address is:${HOST}${data.path}`)
setUrl(`${HOST}${data.path}`)}}})const upload = () = > {
if (fileInput.current) {
fileUploaderClient.uploadFile((fileInput.current as any).files[0])}}return (
<div className="App">
<h1>easy-file-uploader-demo</h1>
<h3>After selecting the file, click the "Upload file" button</h3>
<div className="App">
<input type="file" name="file" ref={fileInput} />
<input type="button" value="Upload file" onClick={upload} />
</div>
{url && <h3>{' file address: ${url} '}</h3>}
</div>)}export default App
Copy the code6-3. Effect of use
First select the large file, and then click Upload. After uploading, the address of the file is displayed.
If you are interested…
We are the Advertising Systems – Interactive technology team, responsible for combining gaming capabilities with advertising to manage and create more interesting and engaging advertising ideas.
Join us and you will face the most complex AD scenarios, the largest data levels, and the most challenging work.
A large number of front-end development & full stack development hc team, Beijing, Shanghai, Hangzhou, Shenzhen have positions, social recruitment, school recruitment, internship is not limited.
Please contact us by email at [email protected].
The first Byte Youth Training Camp – Entering the front end “Registration now”
Click the link to sign up! Mp.weixin.qq.com/s/Pw7Ffi1DN…