Golang is the main development language. This paper mainly introduces the experience of Golang in the process of iteration as well as the framework and library accumulated on Golang.
In the development process, we also accumulated Golang library functions, and based on these library functions to develop the framework and platform, of course, as well as the characteristics of the library, the reason for implementing the library and its advantages.
background
Converged cloud platform – UnionOne cloud started iterative development in 2017. The platform was a private cloud in 2017, able to manage KVM deployed by users on local physical machines as well as bare-metal servers.
At that time, the enterprise IT environment was not only local virtual machines and bare metal, enterprise IT infrastructure has gradually adopted the cloud technology.
So the platform can not only manage virtual machines and bare metals in the local IT environment. IT can also manage other clouds, especially public cloud resources, so that all resources can be managed, o&M and operated in a unified manner on one platform, which can reduce o&M complexity and improve enterprise IT o&M efficiency.
The backend of the platform is Golang, with 600,000 lines of code so far. The front end is Vue framework. The whole platform is based on micro-services framework, and the authentication between each service is based on Keystone components.
Golang accumulation
The first is Golang’s service framework. All components are developed based on this service framework. The characteristics of the service framework are suitable for development on our platform, and optimized according to the characteristics of the platform, which is suitable for rapid development of services.
A Golang service framework and four Golang tool libraries
Jsonutils: JSON serialization and deserialization github.com/yunionio/js…
Sqlchemy: An ORM library that mimics SQLAlchemy github.com/yunionio/sq…
Structarg: structure-based command line argument generation and parsing tool github.com/yunionio/st…
PKG: Some other gadgets and methods github.com/yunionio/pk…
Golang framework
Basically all services are developed based on this service framework, which is a convenient scaffolding framework for CRUD.
Services are mainly operations on cloud resources, such as creating, deleting, and updating cloud resources.
Because there are so many resources in the cloud, scaffolding can be used to facilitate CRUD operations of resources, as well as other mechanisms for complex operations on cloud resources and information recovery.
In addition to CRUD scaffolding, in fact, it adds special features to the platform. First, components are based on Keystone authentication, so when adding Keystone authentication to the framework, the development does not need to pay attention to Keystone authentication, as long as the code is implemented in the framework, naturally integrated with Keystone authentication.
Each API is controlled by permissions, and permissions are integrated into the framework. Every developer developing the platform’s appropriate Restapi does not have to write the appropriate code for permissions and can naturally integrate permissions into the API.
Micro service framework of each service has a corresponding configuration, how to easily manage service configuration, and updated, synchronized to the corresponding components make its effect, this process is relatively complicated, we will service configuration functions are integrated into the framework, the developer USES the framework don’t have to consider the store, update, read the update server configuration and make the configuration take effect, These complexities have been resolved in the framework.
There are also asynchronous task management functions. The platform can be considered as a distributed system, where the cloud controller needs to operate and manage data computing nodes and bare-metal management nodes. Coordinate complex operations between components, such as creating virtual machines and bare metals, which are distributed task management and have an asynchronous task management framework embedded in the platform. This makes it easier to implement asynchronous tasks.
CRUD scaffolding principle
In the platform, each resource, such as the host, corresponds to the MySQL table at the bottom of the database, and the state and corresponding attributes of the resource are recorded in the MySQL table.
Users can operate on the data by calling the API, and at the same time, they can also do additional asynchronous tasks to implement corresponding functions. The underlying code is a resource that corresponds to a MySQL table.
In order to facilitate the operation of MySQL database records, for each resource, a pair of data structures of ModelManager and Model will be corresponding.
ModelManager data structure is the structs corresponding to Golang. This structure can realize the collection operation of this kind of resources, such as creating resources or lists. For the operation of a single resource, Model is used to realize the operation of updating and deleting a certain resource.
Model corresponds to the structure of Golang, which has several fields, each of which represents the attribute of the resource. For example, here there is a user’s resource, the user’s ID, Extra attribute, whether the user is enabled, when the user is created, and the domain to which the user belongs are all corresponding attributes of the user resource.
This attribute is the field of the Golang structure, and through the Tag attribute of the structure field, the definition of the corresponding schema for each field in the MySQL database can be defined.
Charset :” ASCII “nullable:”false” Primary :”true”
This defines that the Id field is a vARCHar (64) field in the database, and its character set is ASCII. Therefore, the tag is used to map the fields of the structure to the fields of the MySQL schema. In this way, each model can be clearly mapped to the MySQL table through the field definition.
If the Model definition and the table definition are not consistent, then the corresponding SQL change operation will be performed to change the table definition and the Model definition to be consistent.
For example, if we change the width of the Id from 64 to 80, we will notice this change when the program restarts, and then change the width of the table to 80.
This allows the Model defined by the code to map exactly to the tables in the database.
Each Model and resource provides a set of apis, and nine classes of apis implemented for a resource have been listed here.
For example, create, delete, update, perform an operation, obtain details, and list.
Each operation corresponds to a RestAPI, and each RESTAPI corresponds to a Manager or Model method for each resource in the back-end code.
For example, the method we need to get the resource details is its restAPI path is GETresources, the Id of resources, and calling this restAPI maps to the GetDetails method of the corresponding Model.
To get the details of the resource, you just need to implement the contents of the GetDetails method in the Model to implement the functionality of the restapi.
The framework simplifies the process of implementing the RestAPI by simply implementing the corresponding Model and ModelManager methods based on the input logic and then returning the correct output.
In this way, surrounding tasks such as completeness, authentication, configuration, synchronization, etc. can be implemented within the framework, greatly improving development efficiency and reducing the chance of making mistakes during development.
There are many things in the framework, including authentication, permissions, configuration change management, quota management, and so on.
Golang library
Here are the Golang libraries that have been accumulated over the course of development. It is conducive to more convenient and efficient implementation of the required functions.
Jsonutils is a library of JSON serialization and antisequencing tools.
Encoding /json is a very powerful and efficient serialization and de-serialization tool library. This is why we need to implement jsonutils library instead of Encoding /json.
Encoding/JSON converts Golang’s data structure to and from the corresponding JSON string.
We can Marshal the Golang struct to generate a Json string, or Unmarshal the Json string to the fields in the corresponding struct to access the struct to retrieve the values in Json.
The obvious difference between JsonUtils and Encoding/JSON is the addition of an intermediate state, which implements the JSONObject in the JsonUtils library.
This is intermediate untyped data, and we can Marshal (s) the data structure into a JSONObject, which is Golang’s interface.
Underneath is a structure where the interface can be further serialized into a JSON string.
This adds an intermediate state between the Golang structure and the JSON string.
This intermediate state is why we need to use jsonUtils. With untyped JsonObjects, we can Marshal (s) any structure into a JSONObject and then pass the JSONObject as a function parameter.
Golang is a strict type checking of language, its each value has the corresponding type, we can handle any API of input and output, the framework of if there is no structure in the middle of the body, when handling API of input and output, the input is a json string, in order to access it in the program, you must to deserialize it into strict type structure, This makes it impossible to turn the framework into a generic framework.
If you take a generic JSONObject, and you put a JSON string in the framework, and you deserialize it into a JSONObject, which is untyped, then you can pass the JSONObject down as a parameter, Until passed to the corresponding method in the specific Model or ModelManager, and then further transformed into the corresponding structure.
This allows the framework to use the deserialized JSONObject and operate on it, enabling a more generic framework.
This is the main reason why we adopted JsonUtils on our platform.
Jsonutils is also special in that JSONObject can be converted not only to JSON strings, but also to QueryString or deserialize QueryString to JSONObject or YAML strings.
This allows for more convenient functionality, such as the API for reading lists like GET, whose arguments are typically QueryString embedded in UR.
We can then deserialize this parameter into a JSONObject in the framework and pass it in as a JSONObject input parameter.
If the parameter is a JSON string in the body, we can also parse it into a JSONObject. We can use the same logic to handle QueryString parameters embedded in UR and Jason parameters embedded in the body.
There’s also something special about JsonUtils, something special about our platform. More important is the support for versioning of structure fields.
Here are some examples:
For example, a structure with an input parameter is called input, and a field is TenantId, which identifies the user’s TenantId.
With the version upgrade, WE hope to change the name of TenantId to ProjectId. If no processing is done in this upgrade, interface compatibility problems may occur. Before the change, the field must be TenantId, but after the change, the field can only be ProjectId. Clients using TenantId cannot access this interface correctly.
In this structure, we added a special tag called Yunion-Deprecated -by, and after upgrading the input structure to the new Input structure, we added a ProjectId field to represent the new TenantId attribute.
The old TenantId is still there, but a Tag named Yunion-Deprecated -by is added to the Tag, and the value of this tag is ProjectId.
Indicates that the TenantId field has been deprecated by ProjectId field.
If the old client has only TenantId, the framework will copy the value of TenantId into the ProjectId field based on the yunion-Deprecated -by tag.
In this way, the ProjectId field can also be used to obtain the value in the new interface even if the old client accesses the new interface.
This ensures that even if you upgrade the interface’s fields, the old client will still be able to access the interface.
This is a feature designed for scenarios where field changes are made to versioning.
Here’s sqlChemy, the library that is the underlying implementation of Model and database mapping in CRUD’s framework.
In SQLChemy, the Golang data structure is strictly unidirectional synchronization to MySQL table, which can strictly generate accurate MySQL schema according to the definition of structure field and the definition of tag in the field.
Ensure that the schema in the database is always strictly consistent with the schema definition. If it is not, you can automatically change the database and then change it to conform to the schema definition.
Another important feature is the ability to implement structured database query statements.
Here is a simple example, we just put the User table for a query, our query in Golang will not implement SQL statements, but will have a structured query method.
The statement in the box above instantiates the UserTable instance and calls its Query interface to return a Query.
Then call q.als (” domain_id a value) as shown in the figure.
This indicates that the user table is being queried, and the value of domain_id is required.
Sqlchemy’s library converts the structure query into a real SQL query at execution time and sends it to MySQL for execution.
The advantage of using this structured approach to database queries is that it avoids the problems of piecing together SQL, and Golang is a strictly static language. The correctness of the query statement can be guaranteed by Golang’s syntax checking.
Another benefit is that by codifying SQL queries, you can reuse some of the database query logic so that you don’t have to repeat the same query statement, but instead call a method that has the SQL query logic embedded in it. SQL Chemy is used in all database operation queries in the framework, which ensures the correctness and execution efficiency of SQL statements in the framework code to a certain extent.
Another important library is structarg, which is used to rigorously map command line arguments or structures in configuration files, information, or code.
So that we can based on the structure to automatically generate the program input parameters, and can according to the values in the command line parameters will be deserialized into the structure, or is the corresponding parameter in the configuration file is deserialized into the structure, can be used in the program access structure of the field to access the value of the corresponding parameters.
This makes it easy to use configuration information in the program. Here’s an example where we define a structure called UserOption, and this structure tag contains each of the fields in each of these structures is a command line, a parameter, or a parameter in a configuration file, The meaning of the Help message is placed in the tag. After initializing the structure and compiling the program, if a Help argument is executed on the command line, it will tell the user the format of the argument in an easy-to-understand way.
At the same time, the user can also use these parameters in the command line arguments to pass the values to the program, in the program code by accessing the structure to access these parameters.
Configuration file input is also supported, and the configuration file supports both KV and YAML formats.
In this way, the configuration information of the program configuration file can be defined by the structure, and the program can automatically identify the configuration file information, and then access the information in the structure.
Framework of configuration management is the basis of structarg function, using the function of each service configuration with defined structure, and configuration information will be put into the database, the database information changes, framework will pull the configuration information in the database, and deserialize it into the structure, The program can sense the configuration change and process the configuration change accordingly.
In addition to the three more important libraries, there are also smaller method libraries in the PKG library.
Making: github.com/yunionio/cl…
Website address: www.yunion.cn/comparison/…