Abstract:

preface

  • Current situation of mobile terminal With the continuous improvement of the hardware level of mobile terminal equipment, now the CPU, memory and other aspects are greatly more than the general PC, so in today’s program, the reasonable use of multi-threading to complete some things is very necessary.

  • Benefits of multithreaded uploads

    1. Further occupy network resources.
    2. Further occupy I/O resources.

Realize the principle of

  • Strategy OSS has the function of sharding upload. Ali Cloud resumable is the encapsulation based on several API interfaces of sharding upload, mainly including InitiateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts.
  • process

  • details
  1. Breakpoint continuation is a large task, and three parts to complete, respectively is to obtain uploadId, fragment upload, upload, this whole continuous steps unified in a thread.
  2. To obtain uploadId, you need to obtain the local cache file first. If you do not get it, you will directly generate a new uploadId and upload it directly. Otherwise, you will restore the record ID of how many pieces were uploaded before and continue uploading at the original position.
  3. In the fragment uploading part, the concurrent uploading mechanism of multi-threading is adopted. At present, the maximum number of threads is 5. Judging by the number of CPU cores, if the number of cores is less than 5, the number of fragments is configured with the number of cores, and the maximum number of fragments is 5000.
  4. After the upload is complete, sort the uploaded parts in the natural order from 1 to N.
  5. File verification: Verifies files based on other information, such as the MD5 information. Md5 verification is performed on each slice uploaded to the server.
  6. The progress callback mechanism, the current progress callback is the most basic version, the current callback principle is based on each fragment back and forth, that is, when the fragment upload success callback.

use

To persist a breakpoint record locally:

android:

String recordDirectory = Environment.getExternalStorageDirectory().getAbsolutePath() + "/oss_record/"; File recordDir = new File(recordDirectory); // Make sure the directory exists. If it doesn't, create itif(! recordDir.exists()) { recordDir.mkdirs(); ResumableUploadRequest request = new ResumableUploadRequest"<bucketName>"."<objectKey>"."<uploadFilePath>", recordDirectory); / / set the upload process correction request. SetProgressCallback (new OSSProgressCallback < ResumableUploadRequest > () {@ Override public void onProgress(ResumableUploadRequest request , long currentSize, long totalSize) { Log.d("resumableUpload"."currentSize: " + currentSize + " totalSize: "+ totalSize); }}); OSSAsyncTask resumableTask = oss.asyncResumableUpload(request , new OSSCompletedCallback<ResumableUploadRequest, ResumableUploadResult>() { @Override public void onSuccess(ResumableUploadRequest request, ResumableUploadResult result) { Log.d("resumableUpload"."success!"); } @Override public void onFailure(ResumableUploadRequest request, ClientException clientExcepion , ServiceException ServiceException) {// Exception handling}});Copy the code

ios:

// Get UploadId to upload, if the task fails and can be continued, Using the same UploadId can upload the same file to store objects on the same OSS OSSResumableUploadRequest * resumableUpload = [OSSResumableUploadRequest new]; resumableUpload.bucketName = <bucketName>; resumableUpload.objectKey = <objectKey>; resumableUpload.partSize = 1024 * 1024; resumableUpload.uploadProgress = ^(int64_t bytesSent, int64_t totalByteSent, int64_t totalBytesExpectedToSend) { NSLog(@"%lld, %lld, %lld", bytesSent, totalByteSent, totalBytesExpectedToSend); }; resumableUpload.uploadingFileURL = [NSURL fileURLWithPath:<your file path>]; NSString *cachesDir = [NSSearchPathForDirectoriesInDomains(NSCachesDirectory, NSUserDomainMask, YES) firstObject]; resumableUpload.recordDirectoryPath = cachesDir; / / record the file path OSSTask breakpoint * resumeTask = [client resumableUpload: resumableUpload]; [resumeTaskcontinueWithBlock:^id(OSSTask *task) {
    if (task.error) {
       NSLog(@"error: %@", task.error);
       if([task.error.domain isEqualToString:OSSClientErrorDomain] && task.error.code == OSSClientErrorCodeCannotResumeUpload) { // This task can not be continued, need to get a new uploadId to upload again}}else {
         NSLog(@"Upload file success");
     }
     return nil;
}];Copy the code

Performance statistics

Data analysis Compared with the previous test after the fragment upload on Android /ios was changed to concurrent, the network request speed of the upload fragment was the same as that of the single thread, which mainly depends on the bandwidth speed. Compared with the single thread, the I/O time of the multi-thread file reading was mainly increased. The data are as follows:

IOS simulator test 100MB size file 1000 Part num single thread 104.530217s Multi-thread 54.528591s 100 Part Num single thread 59.306880s Multi-thread 54.336914s 1.31GB size file 100 Part num Single thread 746.775666s 731.940330s 1000 Part NUM Single thread 822.866331s 733.306236s 2000 Part NUM Single thread 965.428122s 731.940330s 5000 Part Num Single thread 1205.379382s Multi-thread 732.982330s Android motoXT1085 Dual-core CPU 100MB File 100 Part NUM Single thread 70.484s Multithreading 53.656s 1000 Part num Single thread 104.530217s 54.528591s 1.31G Video file 135 Part num Single thread 869s Multithreading 738s 1342 Part num Single-thread 1079.081s Multi-thread 732.076sCopy the code



In general, it is better than before. With the increase of the number of slices, the time of single thread is getting higher and higher. However, under multi-thread, the time is basically the same.

summary

Under the mobile terminal, network resources and I/O resources are generally in short supply, and multi-threading will not improve the total bandwidth of the network: for example, when running a resource download strategy to allocate a connection to supply bandwidth of 2000Kb/s, the local single thread can eat 2000Kb/s at the same time, which reaches a peak value. However, if a resource connection bandwidth is 2000Kb/s, but the single-thread request bandwidth has reached 2000Kb/s, then the local network bandwidth Block upload speed, that is, no matter how many threads, no matter how many connections are also useless; However, if the local network bandwidth has eaten up 2000Kb/s and there are still a lot of network resources left, if there is still room for improvement of 2000Kb/s, then establish a connection to eat up the 2000Kb/s, then the speed can reach 4000Kb/s, then the speed is obvious, The same applies to I/O resources.

The follow-up plan

  • The CRC64 encoding mode is added to verify file correctness, and the server and client conduct interactive verification.
  • The number of threads for fragment uploading is configurable, and users can set the number based on their own requirements.
  • Progress callback optimization, further refinement of the granularity of progress, support configurable callback frequency, etc.