Question the status quo

By creating temporary files on the server and writing the files (folders) into the OutputStream of Response with the help of Hutool’s ZipUtil, the compressed packages of multiple files (folders) can be downloaded. Its general flow chart can be roughly described as:

After analysis and verification, there are the following problems in batch download

  • 1. If a file is very large, step 1.2. 4 Downloading the file to the server takes extra time. For users to download the file, they only need to write the file from the file system.
  • 2. The request type isPOSTTherefore, the browser cannot automatically download files. Step 5 The browser cannot open the download page even if the stream has been written as a responseBlobIn order to open the download, the user experience is very poor, easy to create the illusion of no response to batch download to users.

Is there a way to turn the bulk download interface into a GET request and write the file (clip) directly to the OutputStream of Response?

solution

1. First of all, because of the bulk download interfacebatchDownloadFileIs of the parameter typeList<DownloadFileParam>Is a complex parameter, so it cannot be directly convertedPOSTRequest change to GET; What to do at this point?

In architecture thinking, one of the more commonly used ideas is hierarchical architecture! We can split the batch download interface into two interfaces

Save the download parameter List

to Redis by POST, and return the interface getBatchDownloadKey that uniquely identifies the download parameter in Redis as follows

@PostMapping(value = "/getBatchDownloadKey")
public String getBatchDownloadKey(@RequestBody List<DownloadFileParam> params).Copy the code

The batchDownloadFile interface for batch downloading is defined as follows


@GetMapping(value = "/batchDownloadFile", produces = "application/octet-stream)
public void batchDownloadFile(@RequestParam("downLoadKey") String downLoadKey)
    
Copy the code
2.JavaProvides the classZipArchiveOutputStreamAllows us to directly compress files with directory structure toOutputStream, the pseudo-code used is as follows
ZipArchiveOutputStream zous = new ZipArchiveOutputStream(response.getOutputStream());
//file is a file with a directory structure, such as/folder/subfolder/file.txt
ArchiveEntry entry = new ZipArchiveEntry(file);
InputStream inputStream = file.getInputStream();
zous.putArchiveEntry(entry);
try {
  int len;
  byte[] bytes = new byte[1024];
  //inputStream is a file stream
  while((len = inputStream.read(bytes)) ! = -1) {
    zous.write(bytes, 0, len);
  }  
  zous.closeArchiveEntry();
  zous.flush();
  } catch (Exception e) {
    e.printStackTrace();
  } finally {
    IoUtil.close(inputStream);
  }
Copy the code

This way we can avoid the performance cost of downloading files to the server.

3. The flow chart of the whole process is as follows

Code implementation

Save the download parameter request getBatchDownloadKey

@PostMapping(value = "/getBatchDownloadKey")
public String getBatchDownloadKey(@RequestBody List<DownloadFileParam> params) throws Exception {
    try {
        String key = IdGenerator.newShortId();
        redisTemplate.opsForValue().set(key, JSONObject.toJSONString(params), 60, TimeUnit.SECONDS);
        return key;
    } catch (Exception e) {
        logger.error("getBatchDownloadKey error params={}", params, e);
        throwe; }}Copy the code

BatchDownloadFile is defined as the interface for downloading files based on Key

@GetMapping(value = "/pass/batchDownloadFile", produces = "application/octet-stream; charset=UTF-8")
public void batchDownloadFile(@RequestParam("downLoadKey") String downLoadKey,@RequestParam("token") String token) throws Exception {
    try {
        fileService.batchDownloadFile(downLoadKey, getRequest(), getResponse(),token);
    } catch (Exception e) {
        logger.error("batchDownloadFile error params={}", downLoadKey, e);
        throwe; }}Copy the code

fileService.batchDownloadFile

@Override
    public void batchDownloadFile(String key, HttpServletRequest request, HttpServletResponse response,String token) throws Exception {
        if(redisUtil.get(token) ! =null) {
            UserSession userSession = JSONObject.parseObject(redisUtil.get(token).toString(), UserSession.class);
            // If there is a session or the token is a value that exists in the project_Token configuration, it is authenticated
            if(userSession ! =null) {
                Object result = redisTemplate.opsForValue().get(key);
                if (result == null) {
                    throw new ParamInvalidException("Invalid batch download parameter key");
                }
                List<DownloadFileParam> params = JSONArray.parseArray(result.toString(), DownloadFileParam.class);
                // Create a virtual folder
                String mockFileName = IdGenerator.newShortId();
                String tmpDir = "";
                FileUtil.mkdir(tmpDir);
                ZipArchiveOutputStream zous = null;
                try {
                    // Set the response
                    response.reset();
                    response.setContentType("application/octet-stream");
                    response.setHeader("Accept-Ranges"."bytes");

                    String fileName = URLEncoder.encode(DateFormatUtil.formatDate(DateFormatUtil.yyyyMMdd, new Date()) + ".zip"."UTF-8").replaceAll(\ \ "+"."% 20");
                    response.setHeader("Content-disposition"."attachment; filename*=utf-8''" + fileName);
                    response.setHeader("Access-Control-Expose-Headers"."Content-Disposition");
                    // Parameter assembly
                    zous = new ZipArchiveOutputStream(response.getOutputStream());
                    zous.setUseZip64(Zip64Mode.AsNeeded);

                    DownloadFileParam downloadFileParam = new DownloadFileParam();
                    downloadFileParam.setFileName(mockFileName);
                    downloadFileParam.setIsFolder(1);
                    downloadFileParam.setChilds(params);

                    // Add zip to recursive file streams
                    downloadFileToServer(tmpDir, downloadFileParam, zous);
                    zous.closeArchiveEntry();
                } finally{ zous.close(); }}else {
                throw new ResultException("Internal service error"); }}else {
            throw new ResultException("User offline, please log in again."); }}Copy the code

downloadFileToServer

private void downloadFileToServer(String tmpDir, DownloadFileParam downloadFileParam, ZipArchiveOutputStream zous) throws Exception {
    List<DownloadFileParam> childs = downloadFileParam.getChilds();
    if (EmptyUtils.isNotEmpty(childs)) {
        final String finalPath = tmpDir;
        childs.stream().forEach(dwp -> dwp.setFile(EmptyUtils.isNotEmpty(finalPath) ? finalPath + File.separator + dwp.getFileName() : dwp.getFileName()));
        for (int i = 0; i < childs.size(); i++) {
            DownloadFileParam param = childs.get(i);
            if (param.getIsFolder() == 0) {
                FileInfo fileInfo = fileInfoDao.findById(param.getFileId()).orElseThrow(() -> new DataNotFoundException("File does not exist or deleted!"));
                List<GridFsResource> gridFSFileList = fileChunkDao.findAll(fileInfo.getFileMd5());
                ArchiveEntry entry = new ZipArchiveEntry(param.getFile());
                zous.putArchiveEntry(entry);
                if(gridFSFileList ! =null && gridFSFileList.size() > 0) {
                    try {
                        for (GridFsResource gridFSFile : gridFSFileList) {
                            InputStream inputStream = gridFSFile.getInputStream();
                            try {
                                int len;
                                byte[] bytes = new byte[1024];
                                while((len = inputStream.read(bytes)) ! = -1) {
                                    zous.write(bytes, 0, len); }}finally {
                                IoUtil.close(inputStream);
                            }
                        }
                        zous.closeArchiveEntry();
                        zous.flush();
                    } catch(Exception e) { e.printStackTrace(); }}}// Download files recursively to the compressed streamdownloadFileToServer(tmpDir, param, zous); }}}Copy the code

Project summary

Usually download interface had better use the GET method, the browser will automatically start downloading, in addition, the interface parameters and download interface by adding an intermediary between decoupled to help us solve the problem, POST the download into the GET downloads a layered architecture is software architecture, one of the most commonly used ways to solve practical problems in the process of work, We should be flexible in adopting this method.