preface

This article stems from the unifying problems I encountered while developing and managing many projects and the idea that no matter what development framework or architecture is used, the processing of system logs is a necessary and complex work

First of all, in the development and debugging, whether to console output log content, or export file flow output log content, are relatively distracting work; Secondly, log management is also a vexing problem, such as log storage, log query, log classification and so on

For most small and medium sized systems, it is obviously impractical to include a powerful logging system. Especially for personal projects, SO I searched to see if there was a publicly available log management service that was easy to use, but unfortunately I couldn’t find one

So, this has led to implement a cloud log service idea, purpose is very simple, simply by individual developers RESTful network interface to upload the log, and then through the unified visual management console view logs, as of this release, the log service is implemented, interested can look at the cloud log service UPLOG 】 【

Train of thought

As a log service, it is divided into two parts: upload and view

1. For uploading logs, the most common and simple solution is to use RESTful network interfaces. This allows developers to choose to upload the log stream asynchronously or synchronously across the network

2, for the log view, it is best to provide a visual query system, can be sorted according to the date and log field attributes, so that it can meet the log management needs of most small and medium-sized systems

design

1, upload

A universal cloud log service can be used by different users. Therefore, the core of log storage is partition by user. Besides partition storage, sorting logs by time is also very important. At the same time, the upload interface of UPLOG cloud log service should be as simple as possible and meet data security requirements. Therefore, the upload interface design of UPLOG cloud log service is as follows:

POST https://dl57vasdud645.cloudfront.net/uplog BODY {" sid ", "service identification", "log", "log",... }Copy the code

The interface adoptsapplication/jsonWhere [SID] is the service id provided by the platform, [log] is the required log content, and other fields can be expanded

2, the query

The log system stores a large amount of data, but is sensitive to the query time. The faster developers can query the log content, the greater the development efficiency. This is my experience after using AWS CloudWatch for a long time. The access performance and query convenience of cloud log service provided by Amazon Cloud are both poor and the experience is not good

Therefore, IN the query design of UPLOG cloud log service, I hope it can be as simple and efficient as possible, and the final result is as shown in the figure below (the future will continue to enhance functions and show beautification).

architecture

This cloud log service, I do not want it to be heavy, on the contrary, I want it to be light, but it needs to have strong throughput and scalability, finally I choose to use nodeJS middleware components as the core development of the back-end architecture

1. Nodejs is adopted as the core development language, and the development framework is [UP-KOA]. Koa’s middleware reuse is used to improve development efficiency and speed, and nodeJS’s high asynchronous capability is used for computing processing. 3. Dynamodb is used as the database storage, which is a NOSQL database with no service architecture, featuring adaptive data throughput expansion

Afterword.

UPLOG cloud log service, I finally spent a weekend to complete, and it into the “UPServer cloud application development platform”, UPServer official website preview is as follows

In addition, to be sure it is possible to a lot of people are worried about the logging stored data privacy and security, it is indeed a question worth thinking about, but for UPLOG log upload content is determined by the developers themselves, although UPLOG do the data encryption, but there is no way of data is absolutely 100% security personal guarantee, After all, it’s just a personal interest project. I just thought there should be something that would allow developers to focus more on the core business logic. Moreover, I hope that large cloud platforms such as AWS and Ali Cloud can also launch such application-component cloud services in the future. In my vision of future cloud architecture, developers can freely combine various cloud services just like building blocks, without even writing any line of code. You can achieve a flexible and scalable business system application (which is the vision and philosophy of UPServer)

Thank you for reading and hope you found this article helpful 🙂

Author: CheneyXu

E-mail: 457299596 @qq.com

About: UPServer official website