Template file cloud storage management Sisyphus realizes the upload and download of template files through EasyExcel, and the OSS cloud storage platform stores the template files, and the corresponding file names are stored in the DUCC configuration center. The technical architecture is shown in the figure below.

The core logic

Technology components

EasyExcel

Introduction to the

EasyExcel is a JAVA based simple, provincial saving read and write Excel Alibaba open source project. In the case of as far as possible to save memory support, speaking, reading and writing M Excel, Github:https://github.com/alibaba/easyexcelhttps://github.com/liurenjin/easypoi

Less memory footprint

The main reason why EasyExcel can greatly reduce the memory occupation is that when parsing Excel, it does not load all the file data into memory at one time, but reads the data from the disk line by line and analyzes it one by one. The pattern is parsed row by row, and the result of the parse row is notified to the analysisEventListener in the mode of an observer.

The underlying source

Simple to use

Entity class

@Data public class Hero {@ExcelProperty(value={" 1 "}) private int id; @ExcelProperty(value={" second column "}) private String Name; }

Listening class HeroListener

public class HeroListener extends AnalysisEventListener<Hero> { Logger logger = LoggerFactory.getLogger(HeroListener.class); Private static final int BATCH_COUNT = 100; private static final int BATCH_COUNT = 100; List<Hero> list = new arrayList <>(); // ArrayList = new arrayList <>(); // The first is to provide a UserInfoDataListener constructor. The second is to provide a UserInfoDataListener constructor. This method provides a parameter of type UserInfoDataListener // Alternatively, define the class UserInfoDataListener as an inner class of the UserService implementation class (which is recommended) //private UserService userService; Override public void invoke(Hero Data, AnalysisContext, AnalysisContext) {logger.info(" {}", JSON.toJSONString(data)); list.add(data); if (list.size() >= BATCH_COUNT) { saveData(); List list.clear(); list list.clear(); }} @Override public void doAfterAllAnalysed(analysisContext analysisContext) { Ensure that the last legacy data is also stored in the database saveData(); Logger.info (" All data parsed!" ); } private void saveData() {logger.info("{}});} private void saveData(); , list.size()); // UserService.save (list); Logger.info (" Database saved successfully!" ); }}

The Service layer downloads an Excel

@Override public void testExcelDownload(HttpServletResponse response) { try { List<Hero> stuInfo = this.heroDao.getAllHeroInfo(); response.setContentType("application/vnd.ms-excel"); response.setCharacterEncoding("utf-8"); String fileName = URLEncoder. Encode (" Test ", "UTF-8"); // String fileName = "test "; response.setHeader("Content-disposition", "attachment; filename="+fileName+".xlsx"); EasyExcel. Write(Response.getOutputStream (),Hero.class).sheet(" first sheet").dowrite (stuInfo); } catch (Exception e) { logger.error("testExcelDownload error", e); e.printStackTrace(); }}

Object Storage OSS

Introduction to the

OSS is a distributed object storage service that provides an object storage service in the form of key-value pairs. Users can uniquely retrieve the contents of an Object by its name (Key). Bucket A Bucket is a storage space that a user manages to store objects in. Each user can have multiple buckets. The name of a Bucket must be globally unique within the scope of OSS and cannot be changed once it is created. There is no limit to the number of objects inside a Bucket.

A Bucket is a unit that manages objects for the user. All objects must belong to a Bucket. Buckets have properties that control Region, Object access control, Object life cycle, etc. These properties are applied to all objects under the Bucket, so users can flexibly create different buckets to complete different management functions.

The space inside the same Bucket is flat, that is, there is no file system directory and other concepts, all objects are directly affiliated to their corresponding Bucket.

Object is the basic unit of data storage in OSS, called the Object, also known as the file of OSS. In this article, Object, Object, and file all mean the same thing. Object consists of Meta information, user Data and file name (Key). An Object is identified by a Key that is unique inside a Bucket. The Object Meta information is a key-value pair that represents some of the properties of the Object, such as the last modified time, size, etc. At the same time, the user can also store some customized information in the Object Meta information. Object is immutable throughout the life of the storage. The life cycle of an Object is from the time it is uploaded to the time it is deleted. Repeated uploads of objects with the same name will cause the old Object to be deleted and replaced with a new Object. Therefore, operations such as modifying parts of the file system are not supported in OSS.

AccessKey AccessKey, or AK, refers to the accessKeyID and accessKeySecret used in access authentication. OSS verifies the identity of the sender of a request by using symmetric encryption of AccessKeyID and AccessKeySecret. The accessKeyID is used to identify the user, and the accessKeySecret is the key used by the user to encrypt the signature string and the OSS to verify the signature string. The accessKeySecret must be confidential.

  • The AccessKey applied by the owner of the Bucket.
  • The owner of the Bucket authorizes the AccessKey of the third-party requester through RAM.
  • The owner of the Bucket authorizes the AccessKey of the third-party requester through the STS.

Region represents the physical location of the data center of the OSS. Endpoint indicates the domain name to which the OSS accesses the external service.

​

From a design perspective, mapping OSS to a file system is highly inefficient and not recommended. If you must mount a file system, try to do only write new files, delete files, read files these operations. The use of OSS should give full play to its advantages, that is, mass data processing capacity, and give priority to the storage of mass unstructured data, such as pictures, videos, documents, etc.

OSS Common Operations

Before uploading a file (Object) to OSS, you need to create a Bucket to store the file. Buckets have various configuration properties, including locales, access rights, and other metadata.

Once your Bucket is created, you can upload files of different sizes in a variety of ways.

After the file is uploaded, you can download the file to the browser default path or locally specified path.

When a large number of files are stored in your Bucket, you can optionally list all or some of the files in your Bucket.

Delete Files When uploaded files are no longer needed, single or multiple files can be deleted either manually or automatically by configuring life cycle rules.

DUCC Configuration Center

Introduction to the

DUCC is a configuration center product of JD. It is a new generation of configuration center upgraded on the basis of the original UCC, similar to Apollo. Using long polling pull & timed pull, Relatively open source configuration center has spring – cloud – config, diamond, disconf and Apollo, https://github.com/spring-cloud/spring-cloud-config https://github.com/takeseem/diamond https://github.com/knightliao/disconf https://github.com/ctripcorp/apollo/ Operation mode of DUCC:

  1. On the platform, first create your own application, and then create the corresponding key-value configuration parameters in the corresponding namespace – configuration environment – workspace, and finally publish.
  2. In the project, the corresponding DUCC resource manager is configured, and the connection information of the DUCC platform is added. During the specific use, the configuration is obtained through the KEY.

The characteristics of

  • Multiple environments (or groups) are supported, and groups can be combined
  • Built-in powerful plug-in based data binding framework, support for various types of conversion;
  • Support Log4j, Log4j2, Logback dynamic change diary level function.
  • Support Spring native annotations, support custom annotations, client code intrusion is low
  • Support multi-configuration source client, support for custom configuration, such as ZK, Consol extension
  • Support configuration plan switching

Partial code analysis

Continually update and improve POJO and PojoExcel entity classes

public class Pojo {
}

public class PojoExcel {
}

PojoListener listening class

public class PojoListener extends AnalysisEventListener<PojoExcel> { private static final Logger LOGGER = LoggerFactory.getLogger(PojoListener.class); private List<PojoExcel> dataList; public PojoListener() { } public PojoListener(List<PojoExcel> dataList) { this.dataList = dataList; } @Override public void invoke(PojoExcel Data, AnalysisContext, AnalysisContext) {logger.info (" {}", JSON.toJSONString(data)); dataList.add(data); } @Override public void doAfterAllAnalysed(analysisContext analysisContext) {logger.info (" All data has been analyzed!" ); }}

PojoExcelutil file utility class

public static void writePojoExcel(List<PojoExcel> excelList, HttpServletResponse response, InputStream inputStream) throws Exception{ EasyExcel.write(response.getOutputStream()).withTemplate(inputStream).autoCloseStream(Boolean.TRUE).sheet().doFill(excel List); } public static List<PojoExcel> readPojoExcel(InputStream in){ List<PojoExcel> dataList = new ArrayList<>(); PojoListener listener = new PojoListener(dataList); EasyExcel.read(In,PojoExcel.class, Listener).sheet(" Upload LPS ").doRead(); return dataList; }

The Service logic layer

/ / read the file List < KolTaskAccountHrefExcel > dataList. = pojoExcelUtil readPojoExcel (file. The getInputStream ()); / / write file String filename = duccResourceManager getConfigValue (DuccConstant. POJO_FILENAME); / / filename pojoExcelUtil obtained through DUCC. WritePojoExcel (dataList, response, pojoOssService getExcelFileFromOss (" template file. XLSX "));
Override public InputStream getExcelFileFromoss (String FileName) {StorageObject Object = jingdongStorageService.bucket(excelBucketName).object(fileName).get(); if (storageObject == null) { return null; } return storageObject.getInputStream(); } //<property name=" Excel BucketName" value="${JSS. BucketName. Excel}"/> // JSS. BucketName. Excel = File location in OSS