Download files packed with more than 2GB

This document is downloaded and packaged by the client using streamsaver.js as a “stream”. The company I work for is engaged in blockchain notarization related business, which involves the storage and linking of video, audio, text, PDF and other documents. Therefore, when there are too many evidence files, the notary of the court has to download the evidence one by one, which is not very convenient, so we provide the function of packaging and downloading all of them.

Solution pre-research

FileSaver.js

The popular online version of FileSaver. Js + JSZIP uses a local file download, and then the plug-in exports and packages files stored in the client’s memory. However, there is a big limitation of this plugin is the file size limit, Google Chrome is under 2GB, other lower. Since the memory of the browser is very limited, it is not considered

StreamSaver.js

Streamsaver.js takes the approach of creating a writable stream directly to the file system. Instead of storing data in client storage or memory, for data that is streamed in the browser as you can see from this article handling network requests from a stream perspective is a perfect solution for operations that require storing large amounts of data created on the client side.

With read/write streaming capabilities, one of the problems to be solved is maintaining a continuous channel that allows downloading to be written locally as it goes along. Create a (non-existent) download link and use the Service Worker to block requests for the download link and keep writing to the file stream.

The concept of Service workers

1.Service worker is an event-driven worker registered with a specified source and path. It uses JavaScript to control associated pages or websites, intercepts and modifies access and resource requests, and cache resources in a fine-grained manner. You have complete control over how your application behaves in certain situations, most commonly when the network is unavailable.

2. The Service worker runs in the worker context, so it cannot access the DOM. It runs in a different thread than the main JavaScript thread that drives the application, so it doesn’t block. It is designed to be fully asynchronous, and synchronization apis such as XHR and localStorage are not available in service workers.

3. For security reasons, Service workers can only be hosted by HTTPS. After all, it is very dangerous to expose the ability of modifying network requests to man-in-the-middle attacks. In the User privacy mode of Firefox, the Service Worker is unavailable.

Environmental installation

“Vue2. X Project” “

streamsaver

npm install streamsaver --save
Copy the code

zip-stream.js

It is the core file that handles the download address and launches the following two files

Static resource

Store the following files in the project directory /public at the same level as index.html

  • 1.mitm.html

Forgery the server’s file, which is equivalent to forking a server to download your file and forge a new download address, using the serviceWorker below to write the new download address as a file stream.

File download address: Github download address

  • 2.serviceWorker.js

Process the downloaded content into a written stream

File download address: Github download address

Method implementation

testPage.vue

<template>

 <button @click="downloadAll"></button>
  
</template>
Copy the code

Introduction part

Import streamSaver from 'streamSaver' // import the file we mentioned above import './js/zip-stream.js' // Change the default value in the plugin to local. streamSaver.mitm = 'http://localhost:3010/mitm.html'Copy the code

Method section

methods: {{downloadAll () let data = [[' file 01 ', 'https://xxxx:443.com/xxxx01'], [' file 02 ', 'https://xxxx:443.com/xxxx02']. [' files' 03, 'https://xxxx:443.com/xxxx03']] let urlArr = data. The values / / iteration object let fileStream = StreamSaver. CreateWriteStream (` all files. Zip `) / / packaging of file / * import/zip - stream. After js file window. Zip this method * / let readableZipStream = New window.zip ({pull (CTRL) {const it = urlarr.next () if (it. Done) {ctrl.close()} else {const [name, url] = it.value return fetch(url).then(res => { ctrl.enqueue({ name, stream: () => res.body }) }) } } }) if (window.WritableStream && readableZipStream.pipeTo) { return Readablezipstream.pipeto (fileStream).then(() => {console.log(' download succeeded ')}).catch(()=>console.log' network bad, download failed '))}}}}Copy the code

conclusion

The above problem is a package solution that I haven’t written out in detail, but there are more Examples on StreamSaver’s gitHub for terminating the download

  1. Saving audio or video stream using mediaRecorder
  2. Piping a fetch response to StreamSaver
  3. Write as you type
  4. Saving a blob/file
  5. Saving a file using webtorrent
  6. Saving multiple files as a zip
  7. slowly write 1 byte / sec

I rarely write about this. If there are any problems with this article, feel free to share your thoughts in the comments section