Source: _ blog.csdn.net/ababab12345/article/details/80490621 website: https://idea.markerhub.com

Recently, due to the product needs of the R&D group where I am working, I need to support high performance HTTP upload of large files and request support for HTTP breakpoint continuation. Here’s a quick summary for you to remember:

  1. The server side is realized by C language, instead of using Java, PHP this kind of interpreted language to realize;
  2. The server writes instantly to the disk, so there is no need to call againmove_uploaded_file,InputStreamReaderThis technique requires caching to avoid server footprint and browser request timeouts;
  3. Support for HTML5 and Iframe (for older browsers), and support for getting file upload progress.

In order to better adapt to the current mobile Internet, the uploading service is required to support breakpoint transmission and reconnection. Because the mobile Internet is not very stable; Furthermore, it is very possible to upload a large file with an abnormal drop, so it is necessary to support breakpoint continuation to avoid reuploading.

The idea of supporting breakpoint continuation is as follows:

The client (usually the browser) uploads a file to the server and keeps track of the progress of the upload. In case of an offline or other exception, the client can check the status of a file that has been uploaded to the server and continue uploading from the location of the last uploaded file.

The method is to cut the file into small pieces, such as a 4MB fragment. The server receives a small piece of file each time and saves it into a temporary file. After all the fragments are transferred, the merge is performed. In my opinion, this is fine if the original file is small enough, but once the file is hundreds of megabytes or a few gigabytes or dozens of gigabytes, the time to merge the files can be very long, often leading to browser response timeouts or server blockages.

If you implement your own stand-alone client (or browser’s ActiveX plug-in) to upload files, support for breakpoint continuation is a simple matter of recording the file upload status on the client side. Support for browser breakpoint uploading (no need to install a third-party plug-in) is generally more difficult than doing a stand-alone client upload, but it’s not difficult. My implementation idea is as follows:

1. When the browser uploads a file, it first generates a HASH value for the file, which must be generated on the browser side.

The file upload record cannot be queried solely according to the file name. The repeatability of the file name is very large, and the value composed of file name + file size will reduce the repeatability. If the file modification time is added, the repeatability will further reduce the repeatability conflict. The best way to calculate HASH values is to do an MD5 calculation from the contents of the file, but this is a huge (and unnecessary) calculation, and too much time takes away from the upload experience.

For the above reasons, my calculation of HASH value is as follows:

  1. First, give the browser an ID, which is stored in a Cookie.
  2. The result of browser ID+ file modification time + file name + file size is MD5 to calculate the HASH value of a file;
  3. The browser ID is automatically given to the browser by the system when it accesses the file upload site.
Function setCookie(cname,cvalue, exDays) {var d = new Date(); d.setTime(d.getTime()+(exdays*24*60*60*1000)); var expires = "expires="+d.toGMTString(); document.cookie = cname + "=" + cvalue + "; " + expires; } function getCookie(cname) { var name = cname + "="; var ca = document.cookie.split('; '); for(var i=0; i<ca.length; i++) { var c = ca[i].trim(); if (c.indexOf(name)==0) return c.substring(name.length,c.length); } return ""; } // // Simple file HASH value calculation, if you are not very careful, should be used for production. // Since many types of data are used to calculate file HASH values, the possibility of HASH collisions within the HyFileUploader system should be very small and should be safe to use. // You can use any algorithm to get the file ID, as long as you ensure that the file ID is the same, // function getFileID (file) {// Give the browser a unique ID to distinguish between different instances of the browser (on different machines or from different vendors of the same machine) var clientid = getCookie("HUAYIUPLOAD"); If (cliEnId == "") {// Use a random value as the browser's ID, which will be part of the file's HASH value var rand = ParseInt (Math.random() * 1000); var t = (new Date()).getTime(); clientid =rand+'T'+t; setCookie("HUAYIUPLOAD",clientid,365); } var info = clientid; if (file.lastModified) info += file.lastModified; if (file.name) info += file.name; if (file.size) info += file.size; / / https://cdn.bootcss.com/blueimp-md5/2.10.0/js/md5.min.js var count = md5 (info); return fileid; }

The author thinks that it is not necessary to read the contents of the file to calculate the HASH value, which would be very slow. If you really want to implement HTTP second uploads, you might want to do this, so that if different people upload the same files, you can avoid duplicate uploads and just return the results.

The reason for giving the browser an ID is to further avoid conflicting HASH values for files of the same name and size on other computers.

Mysql > query the HASH value of the file

In support of file uploading, first query file uploading progress information from the upload server through the HASH value of the file, and then start uploading from the upload progress position. The code is as follows:

var fileObj = currentfile; var fileid = getFileId(fileObj); var t = (new Date()).getTime(); Var URL = Resume_Info_URL + '? '; var URL = Resume_Info_URL + '? fileid='+fileid + '&t='+t; var ajax = new XMLHttpRequest(); ajax.onreadystatechange = function () { if(this.readyState == 4){ if (this.status == 200){ var response = this.responseText; var result = JSON.parse(response); if (! Result) {alert(' Incorrect data returned by server, possibly incompatible server '); return; } var uploadedBytes = result.file && result.file.size; var uploadedBytes = result.file.size; if (! result.file.finished && uploadedBytes < fileObj.size) { upload_file(fileObj,uploadedBytes,fileid); } else {// If the file has been uploaded, return the result (result.file);} else {// If the file has been uploaded, return the result (result.file); //var progressBar = document.getElementById(' progressBar '); //progressBar.value = 100; }}else {alert(' Failed to get file breakpoint continuation information '); } } } ajax.open('get',url,true); ajax.send(null);

This is done with the jquery-file-upload component. For the original JavaScript code, see the h4Resume.html sample code in the demos directory.

3. Execute uploading

After querying the file’s breakpoint continuation information, if the file has been uploaded before, the server will return the size of the file that has been uploaded. We can then start uploading data from the size of the file that has been uploaded.

The Slice of the HTML5 File object can be used to cut fragments from a File for uploading.

Definitions and Usage

The slice() method extracts a portion of a word file and returns the extracted portion as a new string.

grammar

File.slice(start,end)

Parameters to describe

The starting index of the fragment to be extracted by start. If it is negative, the parameter specifies the position from the end of the string. That is, -1 refers to the last character of the string, -2 refers to the penultimate character, and so on.

End The subscript of the end of the fragment to be extracted immediately after. If this parameter is not specified, the substring to be extracted contains a string from start to the end of the original string. In addition, pay attention to the public number of JAVA bosom friend, reply to the “back-end interview”, send you a treasure book of interview questions!

If the parameter is negative, it specifies the position from the end of the string.

The code to implement the shard file upload is as follows:

/* fileObj: HTML5 File object start_offset: uploading data relative to the starting position of the File header fileid: */ function upload_file(fileObj,start_offset,fileid) {var XHR = new XMLHttpRequest(); var formData = new FormData(); var blobfile; if(start_offset >= fileObj.size){ return false; } var bitrateDiv = document.getElementById("bitrate"); var finishDiv = document.getElementById("finish"); var progressBar = document.getElementById('progressbar'); var progressDiv = document.getElementById('percent-label'); var oldTimestamp = 0; var oldLoadsize = 0; var totalFilesize = fileObj.size; if (totalFilesize == 0) return; var uploadProgress = function (evt) { if (evt.lengthComputable) { var uploadedSize = evt.loaded + start_offset; var percentComplete = Math.round(uploadedSize * 100 / totalFilesize); var timestamp = (new Date()).valueOf(); var isFinish = evt.loaded == evt.total; if (timestamp > oldTimestamp || isFinish) { var duration = timestamp - oldTimestamp; if (duration > 500 || isFinish) { var size = evt.loaded - oldLoadsize; var bitrate = (size * 8 / duration /1024) * 1000; //kbps if (bitrate > 1000) bitrate = Math.round(bitrate / 1000) + 'Mbps'; else bitrate = Math.round(bitrate) + 'Kbps'; var finish = evt.loaded + start_offset; if (finish > 1048576) finish = (Math.round(finish / (1048576/100)) / 100).toString() + 'MB'; else finish = (Math.round(finish / (1024/100) ) / 100).toString() + 'KB'; progressBar.value = percentComplete; progressDiv.innerHTML = percentComplete.toString() + '%'; bitrateDiv.innerHTML = bitrate; finishDiv.innerHTML = finish; oldTimestamp = timestamp; oldLoadsize = evt.loaded; } } } else { progressDiv.innerHTML = 'N/A'; } } xhr.onreadystatechange = function(){ if ( xhr.readyState == 4 && xhr.status == 200 ) { console.log( xhr.responseText ); } else if (xhr.status == 400) { } }; var uploadComplete = function (evt) { progressDiv.innerHTML = '100%'; var result = JSON.parse(evt.target.responseText); if (result.result == 'success') { showUploadedFile(result.files[0]); } else { alert(result.msg); } var uploadFailed = function (EVT) {alert(" UploadFailed "); ); } var uploadCanceled = function (EVT) {alert(" Upload canceled or browser disconnected!" ); } // Set the timeout time, do not set the timeout time because the upload is large file //xhr.timeout = 20000; //xhr.ontimeout = function(event){// alert(' The file upload time is too long, the server did not respond within the specified time! '); //} xhr.overrideMimeType("application/octet-stream"); var filesize = fileObj.size; var blob = fileObj.slice(start_offset,filesize); var fileOfBlob = new File([blob], fileObj.name); // FormData. append('filename', fileObj.name); FormData.append ('fileid', fileid); FormData.append ('fileid', fileid); FormData.append ('fileid', fileid); // FormData.append ("file", "blob "," fileObj.name "); formData.append('file', fileOfBlob); xhr.upload.addEventListener("progress", uploadProgress, false); xhr.addEventListener("load", uploadComplete, false); xhr.addEventListener("error", uploadFailed, false); xhr.addEventListener("abort", uploadCanceled, false); xhr.open('POST', upload_file_url); // xhr.send(formData); }

In order to verify file breakpoint continuation, the author made a simple interface to display the status information during the process of file uploading. The interface is as follows:

The picture

HTML can be used to calculate the progress of the file upload, the size of the file has been uploaded, the bitrate of the file upload and other information, if there is any abnormality in the upload process, then re-upload can be, the uploaded part will not need to re-upload.

To verify HTML5 Breakpoint Continues, download the file upload server through GitHub and test it.

https://github.com/wenshui200…