In 2007, fed up with C++, Google’s lead software engineer Rob Pike brought together Robert Griesemer and Ken Thompson to create a new language to replace C++ : Golang. Although GO language, which appeared in the 21st century, cannot replace C++ as expected, its near-C execution performance, development efficiency of near-analytic language and near-perfect compilation speed have become popular all over the world. Especially in the cloud project, most of the use of Golang to develop, have to say, Golang has long been popular. For a new project with no historical burden, Golang might be the perfect choice.

Rob Pike, known as the father of GO, said that whether you agree with GO depends on whether you agree that Less is more or lessis Less. Rob Pike encapsulates the entire design philosophy of GO in a very simple way — simplicity and utility.

Many people call GO the C language of the 21st century, because GO not only has the simplicity and performance of C, but also provides a good variety of practical features of server development in the 21st century Internet environment, so that developers can conveniently get what they want at the language level.

Outline of this paper:

• The development and current situation of GO language

• The development history

• The development team

• The business case

• Key features of the GO language

• Concurrency and coroutines

• Communication based on message passing

• Rich and useful built-in data types

• The function returns multiple values

• Defer Defer processing mechanism

• Reflection (reflect)

• High-performance HTTP Server

• Engineering management

• Programming specification

• API rapid development framework practices

• Why did we choose GO

• Implementation of API framework

• Common component capability

• Generic list component

• Generic form component

• Assist in fibers

• Data validation

• summary

• The performance evaluation

• Points to note during development

The development and current situation of GO language

The development history

In September 2007, Rob Pike was compiling C++ on Google’s distributed compilation platform. During the long wait, he and Robert Griesemer discussed some of the key issues of programming languages. They argued that simplifying a programming language rather than adding new features to a bloated language, It’s going to be a bigger step forward. They then convinced Ken Thompson that something needed to be done before the build was finished. A few days later, they launched a project called Golang as a free-time experiment.

In May 2008, Google saw the potential of GO and got the full support of Google. These people started to work on the design and development of GO full time.

The first version of GO was released in November 2009. The first official release, Go1.0, was released in March 2012.

Go1.5 was released in August 2015, and this release was considered historic. Completely remove the C language part, use GO to compile GO, a small amount of code using assembly implementation. In addition, they brought in Rick Hudson, a leading expert on memory management, to redesign the GC to support concurrent GC, and solve the long-standing problem of GC latency (STW). In later versions, GC was further optimized. By GO1.8, GC latency in the same service scenario can be reduced from a few seconds in GO1.1 to less than 1ms. With the resolution of the GC problem, it can be said that the GO language eliminates almost all of the weaknesses in server-side development.

During the iteration of the GO language, the features of the language have not changed much, basically maintaining the benchmark of GO1.1, and the official promise is that the new version is fully compatible with the code developed under the old version. In fact, the GO development team has been cautious about adding new language features and has continued to optimize in terms of stability, compile speed, execution efficiency, and GC performance.

The development team

GO language development camp can be said to be unprecedented strong, the main members of the computer software industry is no lack of historical figures, the development of computer software far-reaching influence. Ken Thompson, from Bell LABS, designed THE B language and created the Unix operating system (originally implemented in B). Later, during the development of Unix, he and Dennis Ritchie designed THE C language and rebuilt the Unix operating system using C. Dennis Ritchie and Ken Thompson are known as the fathers of Unix and C and were jointly awarded the Turing Award in 1983 for their contributions to the development of computer software. Rob Pike, also of Bell LABS and a key member of the Unix group, invented Limbo and co-designed utF-8 encoding with Ken Thompson, co-authors of Unix Programming Environments and Programming Practices.

It can be said that GO language back to Google this tree, and there is no lack of cattle, is worthy of the name of the “second generation of cattle”.

The famous Docker is completely implemented with GO, the most popular container choreography management system kubernetes in the industry is completely implemented with GO, and the later Docker Swarm is completely implemented with GO. In addition, there are various well-known projects such as ETCD, Consul, Flannel, etc., which are implemented using GO. Some people say that the GO language is famous for catching up with the cloud era, but why not put it another way, it is also GO language to promote the development of cloud?

In addition to cloud projects, there are companies like Toutiao and UBER that have completely restructured their businesses using GO.

 

Key features of the GO language

GO language is powerful, because it in the development of the service side, always can grasp the pain point of programmers, in the most direct, simple, efficient, stable way to solve the problem. Instead of going into the specifics of GO’s syntax, we’ll introduce you to the key aspects of the language that are important for simplifying programming, and follow in the footsteps of the great masters and experience the design philosophy of GO.

Key features of GO include the following:

• Concurrency and coroutines

• Communication based on message passing

• Rich and useful built-in data types

• The function returns multiple values

• Defer mechanism

• Reflection (reflect)

• High-performance HTTP Server

• Engineering management

• Programming specification

 

In today’s multi-core era, concurrent programming makes sense. Of course, many languages support multithreaded, multiprocess programming, but unfortunately, it’s not that easy and pleasant to implement and control. Golang differs in that the language level supports goroutine concurrency (also known as microthreads, which are lighter, cheaper, and perform better than threads) and is very simple to operate. The language level provides keywords (go) to start coroutines, and thousands of coroutines can be started on the same machine.

Contrast JAVA multithreading and GO coroutine implementation, obviously more direct, simple. This is the charm of GO. It can solve problems in a simple and efficient way. The keyword GO is perhaps the most important symbol of GO language.

Communication based on message passing

In asynchronous concurrent programming, it is not enough to start coroutines easily and quickly. Message communication between coroutines is also very important, otherwise, individual coroutines will become wild horses and get out of control. In GO, message-based communication (rather than the shared memory communication used in most languages) is used for intercoroutine communication, and message channels are used as basic data types, defined by type keywords (CHAN), which are thread-safe for concurrent operations. This is also revolutionary in the implementation of language. It is clear that GO language itself is not so simple that there is no bottom line, it is precisely they will be the most practical, most conducive to solving problems, in the simplest and most direct form to the user.

Channel is not only used for simple message communication, but also extends to many very practical and convenient functions. For example, implementing TCP connection pooling, limiting traffic, and so on are not easy to implement in other languages, but GO is easy to do.

As a compiled language, GO language also supports very comprehensive data types, in addition to the traditional integral type, floating point type, character type, array, structure and other types. As a matter of practicality, there is also native support for string types, slice types (variable length arrays), dictionary types, complex types, error types, pipe types, and even arbitrary types (Interface{}), and it is very easy to use. Strings, slice types, for example, are almost python-like in their simplicity.

In addition, error is used as the base data type, and try is no longer supported at the language level… The use of “catch” is a bold and revolutionary innovation, and it is no wonder that many people ridicule the language of “GO” as neither fish nor fowl. But out of the traditional ideas, developers think of GO in the programming process, to ensure the robustness and stability of the program, high-precision processing of anomaly is very important, only in each logical processing is completed, clear told the upper calls, whether there are abnormal, and the upper clear call, in a timely manner to deal with abnormal, Only in this way can the robustness and stability of the program be guaranteed to a high degree. While this can lead to a lot of errors in the programming process, it also makes developers more vigilant about exception handling. As it turns out, it’s hard to write unrobust code as long as you code strictly in the style recommended by GO. That is, of course, if you don’t reject it and approve of it.

 

Support for multiple return values for functions is nothing new in languages, and Python is one of them. Allowing functions to return multiple values can simplify programming in some scenarios. The programming style recommended by the GO language is that the last parameter returned by a function is of type error (whenever an exception is possible in the logic body), so it is necessary to support multiple return values at the language level.

 

Defer Defer processing mechanism

In the GO language, you provide the keyword defer, which allows you to specify the logical body that you want to defer, either before the return of the function body or when panic occurs. This mechanism is well suited for aftercare logic, such as avoiding potential resource leaks early on.

It’s safe to say that defer is another very important and useful language feature after Goroutine and Channel. The introduction of defer greatly simplifies programming and makes the language description more natural, greatly increasing the readability of the code.

 

Golang, as a strongly typed compiled language, is naturally less flexible than a parsing language. PHP, for example, is weakly typed and can directly new the contents of a string variable, which is obviously not possible in compiled languages. However, Golang offers the Any type (interface{}) and powerful type reflection capabilities, which, combined, make development very close to analytic. In terms of dynamic invocation of logic, it is still very simple to implement. So what’s the advantage of an analytic language like PHP over GO? Personally, I have written PHP for nearly 10 years and realized the development framework, basic class library and various common components. Although the execution performance is insufficient, the development efficiency is more than enough. When it comes to Golang, those advantages don’t seem so obvious.

 

As a server-side language appearing in the Internet era, the ability of user-oriented service is essential. GO has its own HTTP/TCP/UDP high-performance server at the language level. Based on coroutine concurrency, it provides the most direct and effective capability support for business development. Implementing a high-performance HTTP Server in the GO language is as simple as a few lines of code.

 

In GO, there is a standard project management specification that makes everything (package management, compilation, etc.) much easier once you follow it.

Under the GO project, there are two key directories, one is the SRC directory, which is used to store all the. GO source files; One is the bin directory, where compiled binaries are stored. In the SRC directory, except the directory where the main package resides, all directory names must correspond to the package names in the direct directory. Otherwise, the compilation fails. This way, the GO compiler can start from the directory where the main package is located and use the directory structure and package name entirely to derive the project structure and build order, avoiding introducing an extra Makefile as C++ does.

During GO compilation, the only thing we need to do is assign the GO project path to an environment variable called GOPATH to let the compiler know where the GO project to compile is located. Then go to the bin directory and run go build {directory name where the main package resides} to complete the project compilation in seconds. Compiled binaries can be pushed to the same OS to run directly, without any environment dependence.

The programming specifications of GO enforce integration into the language, such as specifying the placement of curly braces, enforcing a one-line rule, not allowing you to import unused packages, not allowing you to define unused variables, providing gofmt tools to enforce formatting code, and so on. Oddly enough, this has also caused a lot of dissatisfaction among programmers, with some publishing XX complaints about the GO language, including accusations about programming specifications. Remember, from an engineering management perspective, any development team will have specific programming specifications for a particular language, especially at a company like Google. GO’s designers felt that rather than having the specification written in documentation, it would be more straightforward to force integration into the language, which would take advantage of teamwork and project management.

API rapid development framework practices

A programming language is a tool that tells us what we can do, and how we can do it better. This section introduces a development framework implemented in the GO language, as well as several common components. Of course, frameworks and common components can be implemented in other languages, but the concern here is cost. In addition to the GO language itself, we hope to give you some idea of how to solve a problem in some way, rather than just writing code to solve a problem at the end of the day. If you accept this approach, I believe that the following may influence the way you develop your future projects, and radically improve the efficiency of your development.

Why did we choose GO

GO was chosen for two main reasons

1. performance

   Shorten the API response time to solve the problem of batch request access timeout. In the business scenario of Uwork, a batch REQUEST of API often involves multiple calls to other interface services. However, under the previous PHP implementation mode, it is very difficult to achieve parallel calls, and serial processing cannot fundamentally improve the processing performance. But GO language is different, through the coroutine can easily realize the PARALLEL processing of API, to maximize the processing efficiency.

   Relying on Golang’s high-performance HTTP Server, the system throughput capacity has been improved from hundreds of LEVELS of PHP to thousands of miles or even more than ten thousand levels.

2. Development efficiency

   GO language is simple to use, high efficiency of code description, uniform coding specification and quick to use.

   With a small amount of code, you can standardize the framework and quickly build API business logic with uniform specifications.

   It can quickly build various common components and public class libraries to further improve development efficiency and achieve mass production of functions in specific scenarios.

 

When learning a new language or starting a new project, many people have a habit of looking for an open source framework online to start their project. There’s nothing wrong with that, but I personally think it’s more helpful to know the internal implementation. Perhaps you have noticed that the MVC framework, in essence, is to parse the request path, and then route to the corresponding controller (C) according to the request path segment, and then the controller further calls the data logic (M), gets the data, renders the view (V), and returns to the user. The core of the whole process is the dynamic invocation of logic.

However, the implementation of the API framework is simpler than the implementation of the WEB page framework, because it does not involve rendering the view, just returning the data results to the user in a protocol manner.

Use GO language to implement a complete SET of MVC development framework, it is very easy to integrate HTTP Server at the same time, the entire framework of the core code is not more than 300 lines, from here can actually feel the GO language description efficiency is high (if interested, please refer to the Uwork open source project Seine).

Others say that in GO, there is no framework to speak of, and the implication is that introducing a heavy open source framework is not necessary and may complicate simple things.

 

In the actual project development process, only the efficient development language is not enough, in order to further expand the development efficiency, continuous precipitation of common base library is essential, in order to further abstract and reuse the common base logic.

In addition, the common component capability is the fundamental to achieve the mass production of functions, which will be a qualitative improvement to the development efficiency. Componentized development helps us improve problem solving from one point to another. The following highlights the implementation of a few common components that can truly liberate programmer productivity. These powerful common components are not complicated to implement in Golang. At the same time, combined with Golang’s concurrent processing capability, compared to the PHP version of the implementation, the execution efficiency will also be improved. This is the perfect combination of component capability and language efficiency.

 

The generic list component is used for data query scenarios with all possible two-dimensional data sources (such as MySQL/MongoDB/ES, etc.) and solves the data query problem from one side. In the development of Uwork project, it has been widely used to realize the mass production development of data query interface and page query list. It centers around a JSON configuration file to query a common data source and automatically returns the query results to the user in the form of an API or a page. There was almost no code development, and all that was required was to write configuration files (rather than code) in a unified specification, which really enabled the mass production of functionality for data query requirements.

This is the process of building a generic list component. Does it feel out of reach to implement such a powerful generic component? That’s not the case, and once you figure out how it works, it’s not that complicated to incorporate your building ideas into Golang. In our project, the implementation of the entire component solved a series of data query problems in less than 700 lines of Go code. In addition, the parallel execution of the field processor is realized through the concurrent feature of Golang, which further improves the execution efficiency of the component. It can be said that the fusion of Generic lists and Golang is the perfect combination of performance and efficiency.

 

The common form component is mainly used for adding, deleting, and modifying databases. This component is also widely used in the development of Uwork projects. Similar to the general list, it takes a JSON configuration file as the center to complete the operation of adding, deleting and modifying data in the data table. In particular, the recently completed component-level SDB management platform realizes the data maintenance of the whole system through common forms, and achieves the code-free production of business through high abstraction.

 

This is the complete build process for a generic form, but for this one component implementation, we solved the whole aspect of maintaining data on a data table in less than 1,000 lines of GO code.

The GO language itself supports coroutine concurrency, and coroutines are very lightweight and can quickly start thousands of coroutine units of work. If the number of coroutine tasks is not properly controlled, the end result is likely to be counterproductive, resulting in unnecessary stress on external or internal services. Coroutine pool can control the number of execution units to a certain extent and ensure the security of execution. Implementing such a coroutine pool in Golang is very simple, requiring only a little encapsulation of channels and Goroutine, and the entire construction process is less than 80 lines of code.

Data validation is always an integral part of API development. A few lines of code may be enough for simple data validation, but hundreds of lines of code may not be enough for complex data validation, especially for recursive data validation, which is a nightmare.

The data verification component can be configured using a data template to complete the general verification using specific logic. The developer only needs to configure the corresponding data template and make a simple call to complete the verification process. For such a general-purpose data validation component, the entire build took less than 700 lines of code in GO.

summary

In the actual project development process, the biggest improvement in development efficiency is undoubtedly the common component capability that fits the business scenario of the system. This is also in line with Rob Pike’s saying (Less is Lessor Less is more) that the truly efficient development is configured and does not require much code. The logic can be implemented without writing any code at all, and this way is optimal for maintenance costs because of the high degree of uniformity.

The efficiency of GO’s language description is undeniable, and the implementation of all of the common components mentioned above solved a problem in one area in less than 1,000 lines of code.

(Some of the above code is already available in the Uwork open source project Seine)

 

The performance evaluation

Description of pressure test environment:

• Service running machine: single idle B6, 24 core CPU, 64G memory.

• PHP API environment: Nginx+PHP-FPM, CI framework Nginx starts 10 child processes, each process can receive a maximum of 1024 connections. Php-fpm uses static mode to start 2000 resident child processes.

• Golang API environment: compiled with go1.8.6, directly pull up the Golang API Server process (HttpServer), regardless of tuning.

• Client initiated request test program: written using Golang, coroutine concurrency, run on an independent spare B6, 24 core CPU, 64G memory, request 20000 times on 1-2000 different levels of concurrency (concurrency step length is 50).

 

Comparison of stress test results

 

In the Golang API framework, the processing QPS fluctuates around 6.5 W /s when the number of concurrent requests is greater than 50. Stable performance, no error in the pressure test process.

Nginx+php-fpm, only prints exit(‘ OK ‘) in index.php, handles QPS fluctuation around 1w/s when concurrency >50. Stable performance, no error in the pressure test process.

In Nginx+ PHP-Fpm +CI framework, logic executes to specific business logic point, output exit(‘ OK ‘), handle QPS around 750/s when concurrency >50. And the performance is unstable, the number of errors increases with the increase of concurrency in the process of pressure test.

Stress tests show that Golang and PHP are not comparable in terms of performance; However, the HTTP API framework implemented by Golang can achieve a single QPS of 6.5 W /s when no load, which is very satisfactory.

Points to note during development

The following are some problems encountered in the actual development process for reference only:

1. Do not use panic/recover to simulate throw… Catch, that’s what I did at first, but it turned out to be totally self-righteous.

2. Native Error is too simple, and in actual API development, different exception cases require different return codes, so it is necessary to encapsulate error again.

3. The execution body of any coroutine logic must defer recover() at the beginning of the logic; otherwise, the panic in Goroutine will cause the whole process to break down. It is necessary to avoid the global impact caused by some logic bugs.

4. In Golang, operations on variables (except chan) are non-thread-safe, including basic types like int, so it’s important to consider locking when concurrently operating on global variables, especially on maps.

5. All map key values should be evaluated for existence, and it is best to uniformly encapsulate similar operations to avoid unnecessary runtime exceptions.

6. When defining slice data types, try to preset lengths to avoid unnecessary internal data reorganization.