Server-side development generally refers to the writing of business interfaces. For most systems, CURD operations account for the majority of interfaces. However, there are always online memes poking fun at “CURD engineers” to show that such development techniques are not complicated. However, IN my opinion, it is indeed simple just to find a framework to fill the point code to complete the task, but human beings are a “thinking reed”, if you think deeply, there will be many common problems in the development process. Take the go framework, for example. It comes in two different forms: a type of framework represented by Beego, which GoFrame continues to promote, is characterized by being large and comprehensive, offering a wide variety of features, and you don’t even have to choose much, just follow the documentation. The problem with them is that many times because they are so well encapsulated, many of the problems have already been solved intangibly (but not necessarily in the best way). On the other hand, frameworks such as GIN and Go-Mirco solve only a specific part of the problem, and there is a lot of additional work to be done with them, but there is also a lot more to be learned from them. Next, let’s take a closer look at some of the possible problems with go server-side development:

1. Project structure

Whether it is a large project or a small management system, the first step in the long march is how to organize their own project structure. In terms of project structure, GO actually has no fixed criteria, so it can be organized flexibly according to the actual situation. But I think there are a few things to be aware of:

1. The package name is simple, but pay attention to the meaning of the name

This has already been mentioned in this article, using short abbreviations instead of long package names, and common abbreviations like FMT, Strconv, PKG, CMD, etc. But I think it’s more important to know what you mean than to be simple. For example, I was working on a project that had an MDW package in its root directory. I didn’t know what it was for until I saw some gin middleware in it, which was an abbreviation for Middleware. So although go officially recommends some conventional, concise package names, there should be a precondition, which is to explain the function of the package in the comments, which are very lacking in the domestic environment. So rather than making up abbreviations and not commenting, it’s better to write the package name clearly.

2. Use internal

Using internal helps force people to think about what should be in the public package and what should be in the private package, making the project structure clearer. Go itself provides less detailed package access than Java, with only two states, public and private, which should be supplemented by internal.

3. Don’t use init casually

To be honest, I have some doubts about why init isn’t restricted at all, which means that some of the libraries you rely on can run before your program code and you don’t know what it does (any code can be executed in init). This is evident in large projects where there are a lot of dependencies and a lot of indirect dependencies. Although go officially requires that no complex logic be performed in init, this is not binding. The simplest example is unit testing. Sometimes I run unit tests and panic fails to run. The reason is that some dependent library init does some dirty operations. But here’s the thing: I rely on the library indirectly, and I have no control over its code. In this case, it can only continue to run if it completes its requirements in the unit test. So think twice about putting code in init. From what I can see, a lot of init code does do initialization, but it implicitly depends on files, paths, resources, etc. In this case, consider whether the NewXX() \ InitXX() function can be used instead.

4. Use package names like util \ common cautiously

This is often used by Java programmers, but in go, meaningful package names are recommended instead of meaningless package names. For example: util.newtimehelper () is not good and should be written as time_helper.new () to make it more readable. But I think the specific situation has to be analyzed on a case-by-case basis, so the title is caution, not no use. Because sometimes, your util \ common is just a few helper functions and not much. Breaking it down into a few packs feels like a bit of a loser, so I’ll wait for Util \ Common to save a bit more and then rebuild. So back to the beginning, think and be flexible. However, if it is a public util \ common that many people rely on, it is better to break it up early, or it may not be able to break it up later.

2. Code structure

Code structure can say more things, this may be to see the foundation of software design, I am also a beginner in this respect, so summed up may be right, may not be right, just for reference.

1. Division of C \ S \ D layer

Thanks to the popularity of MVC, most projects still have internal partitions such as ControllerOrHandler, ServiceorSvc, daoorRepository, even though the front and back end separation is now common. Logic used to isolate data presentation, logic processing, and data access. Here is my understanding of these three layers: Controller: generally, it is the entry controller, which is responsible for parameter reception, conversion, return value processing, protocol processing and other work. This layer is usually not too thick, meaning there is not much logic. Note the difference with a Gateway, which does and can do much more. In my opinion, parameter verification should be done by using some framework such as Validator instead of repeating the wheel. If you need to access the database to verify parameters, it should be done in the Service layer. Service: This layer can be quite heavy, and it is also a test of design skills. If you are not careful, it is easy to make this layer highly coupled. I’ve also seen SQL queries written directly in the Service layer become a pain in the neck. In general, because this layer connects, try to make it a glue, not an all-rounder. Dao: This layer is related to data. In fact, it is the service layer’s direct operations on data (operation database, redis) into method calls. To shield database differences, but also can do some unified data processing. This layer also encapsulates the ORM to make it easier to use. Since this layer is more of a generalization of the data, it is usually more convenient to generate from a code generator, such as GORmt.

2. Dependency passing

The dependency here refers to the dependencies of controller, Service and DAO. Generally speaking, controller needs to call Service, and Service needs to call DAO. The worst thing to do is to call the code that creates the lower layer because the upper layer needs the lower layer, such as NewService in the controller constructor (i.e., NewXX; there is no special constructor in Go), which clearly does not meet the design principle of a single responsibility. So there are generally two ways to deal with: first, set up global variables

var XX *XXService = &XXService{}

type XXService struct{}func (x *XXService) XX(a){}Copy the code

This way, other layers can call the global variable directly. Convenience is convenience, but also easy to bring two problems: 1. Arbitrary call so that not only the upper layer can be called, the lower layer can also be called. This can be done all over the place, and it can easily become unmanageable, especially on projects with multiple people working together. 2. Holding a field in an XXService must involve initialization. If you place a field in an init, as mentioned above, it is not a good idea to place it in an init. If you set a NewXX() function, you do not need to set this global variable. 2. Set up the NewXX() function, managed through the dependency injection framework

type XXService struct{
	xRepo XXRepo
}

func NewXXService(r *XXRepo) *XXService {
	
}
Copy the code

These constructors are then managed through a dependency injection framework

// wire.Build(repo.NewGoodsRepo, svc.NewGoodsSvc, controller.NewGoodsController)
// The wire framework is automatically generated
func initControllers(a) (*Controllers, error) {
	goodsRepo := repo.NewGoodsRepo()
	goodsSvc := svc.NewGoodsSvc( goodsRepo)
	goodsController := controller.NewGoodsController(goodsSvc)
	return goodsController, nil
}
Copy the code

Here, the Wire framework is nowhere near as cool as the dependency injection framework in Java, and in fact, saves us the trouble of writing our own. So the Controller holds the Service object, and the Service holds the Repo object. Moreover, only registered can hold, avoiding the problem of administrative confusion.

3. Avoid global variables

When it comes to the problem of global variables, it is necessary to take them out in detail. The most typical example of a global variable is logger. As we all know, the log package provided by Go is not very easy to use, so we usually use some open source logger implementation. Many implementations will provide a default log (defaultLogger), which is easy to use. But because go itself advocates in-place error handling, that is, familiarity with if ERR! = nil, that’s the problem. In some large projects, there may be many functions, library writers, after judging the error, handy to print out a log; Or printing debug logs all over the place. As a result, when we look at the program log, we may see a large number of spam logs, such as: an error is printed multiple times; Or it’s full of useless crap. The number of logs per day may be as high as a dozen gigabytes, do not know how busy you think business, in fact, all nonsense log. Zap doesn’t have this global Logger. You have to create a New object, which forces you to think, how do you save this object? How do I get it to where I need to print the log? Unfortunately, I’ve also seen zAP log objects assigned to global variables and used like crazy. This is the same problem that occurs when using GORm, throwing the returned Gorm. DB object into a global variable and then using it around. This is the same problem that occurs when using Logger.

3. Observable processing

Observability refers to the organic combination of logging, link tracing, and monitoring. For details, see the references in the appendix. Observability was also known to the public after the popularity of Jaeger and Prometheus. Although the history is not too long ago, its importance is self-evident. To effectively solve problems in online services, it is very necessary to quickly find and locate specific locations, whether it is a large project, a small project, a micro-service architecture or a single architecture. In terms of specific coordination, monitoring services such as Prometheus can be used to detect service anomalies in time, and Jaeger +log can be used to find the context when problems occur, so as to quickly locate faults. Before I did not use link tracking, monitoring and other technologies, only log, many online errors can not be found in time or cannot be repeated, in fact, it is a pity. However, in order to achieve observability, some code needs to be modified, and completely non-invasive modification is not possible, so it can be considered in the design phase of the project. In terms of invasiveness, monitor < link trace < log. For example, to access Prometheus, a sidecar thread should be run to handle the pull requests of Prometheus, but the program should also have the code to collect monitoring indicators written in advance. Link tracing is more in-depth to each layer of the project, such as: controller->service-> DAO -> DB, so as to track the whole request link; Logs must be the most intrusive, as they are printed in business code.

1. Db \redis\log link tracking processing

The core of link tracing is context. Generate a tracing context at the beginning of the request, and each layer processes and passes the context. If it is code inside the project, parse out the span from the context and print data into the SPAN. However, for some libraries that the project depends on (gorm\zap\redis, etc.), if you want to trace links to the inside of these libraries, there are two ways to do this:

  1. The gorm library itself supports passing the context, such as: gorm can pass the context in, although it does not help you parse the context, but provides hook capabilities, you can write a plugin to get the context, their own processing can be. Or a framework like Go-Micro, which automatically handles the context parsing.
/ / gorm example
	// Use plugins
	err = db.Use(NewPlugin(WithDBName(dbName)))
	iferr ! =nil {
		return nil, err
	}
	
	/ / query
	DB.WithContext(ctx).Find()
Copy the code
  1. The library does not support passing a context, or the library does support passing a context, but does not provide hook capability. Since we can’t modify the library’s code or hook its internal key operations, we need to take over the access to the library through proxy mode, such as go-redis
type Repo interface {
	Set(ctx context.Context, key, value string, ttl time.Duration, options ... Option) error Get(ctx context.Context, keystring, options ... Option) (string, error)
	TTL(ctx context.Context, key string) (time.Duration, error)
	Expire(ctx context.Context, key string, ttl time.Duration) bool
	ExpireAt(ctx context.Context, key string, ttl time.Time) bool
	Del(ctx context.Context, key string, options ... Option)bool
	Exists(ctx context.Context, keys ...string) bool
	Incr(ctx context.Context, key string, options ... Option)int64
	Close() error
}

type cacheRepo struct {
	client *redis.Client
}
Copy the code

CacheRepo is a proxy that holds a private redis.Client object to facilitate link tracing during Set and Get operations. Here is an example of the Get method:

func (c *cacheRepo) Get(ctx context.Context, key string, options ... Option) (string, error) {
	var err error
	ts := time.Now()
	opt := newOption()
	defer func(a) {
		ifopt.TraceRedis ! =nil {
			opt.TraceRedis.Timestamp = time_parse.CSTLayoutString()
			opt.TraceRedis.Handle = "get"
			opt.TraceRedis.Key = key
			opt.TraceRedis.CostSeconds = time.Since(ts).Seconds()
			opt.TraceRedis.Err = err

			addTracing(ctx, opt.TraceRedis)
		}
	}()

	for _, f := range options {
		f(opt)
	}

	value, err := c.client.Get(ctx, key).Result()
	iferr ! =nil {
		err = werror.Wrapf(err, "redis get key: %s err", key)
	}
	return value, err
}
Copy the code
2. The middleware

Middleware refers to a tripartite framework that extends go’s native HTTP framework to support a chain of request methods, such as Gin, Negroni, etc., so that we can insert processing logic before and after a request, such as Panic-Recover, authentication, etc. Speaking of the need to generate the tracing context at the beginning of the request, this can be done in the middleware (in the case of the microservice architecture, it should be generated at the entry gateway). The generated tracing context can be directly transmitted to the log, so that all the logs printed in the subsequent request link are equipped with TraceID for easy tracing. At the same time, request monitoring metrics (QPS, response time, etc.) can also be put together in this middleware.

4. Error handling

1. Response Error processing

In general, our interface returns are used to returning an error code to handle some business logic errors. This error code is different from the HTTP status code, generally is a self-defined error code table, but for the sake of standards, we still need to return compatible with some commonly used HTTP status code (400, 404, 500) and so on. In this way, our Response Error needs the following capabilities:

  1. Hide program errors, especially panic errors, once thrown, easy to analyze the internal implementation details of the system, so pay attention to hiding. In particular, many Web frameworks will automatically recover panic and print it out.
  2. You can easily define HTTP status and error codes by specifying the return status and error codes at the Service layer, since only the Service layer has global control.

It is very simple to implement, just implement the following five methods:

// Create an Error based on the status code, Error code, and Error description
func NewError(httpCode, businessCode int, msg string) Error {}
// The default status code is 200. Create an Error based on the Error code and Error description
func NewErrorWithStatusOk(businessCode int, msg string) Error {}
// The status code defaults to 200. Create an Error based on the Error code (the Error description is obtained from the Error code table).
func NewErrorWithStatusOkAutoMsg(businessCode int) Error {}
// Create an Error based on the status code
func NewErrorAutoMsg(httpCode, businessCode int) Error {}
// Put the internal err in the Error
func (e *err) WithErr(err error) Error {}
Copy the code

The Error structure encapsulates the status code, Error code, Error description, and true Error. When used, the sample code is as follows:

func (s *GoodsSvc) AddGoods(sctx core.SvcContext, param *model.GoodsAdd) error{...iferr ! =nil{
		return response.NewErrorAutoMsg(
			http.StatusInternalServerError,
			response.ServerError,
		).WithErr(err)
	}
Copy the code
2. Error handling of Go

Now that I have written error code, I have to mention the error handling in Go. Error handling has always been a big controversy in Go. In terms of our daily development, there are three problems most commonly encountered, and the Go official only solved one of them in version 1.13.

  1. Since errors can actually only contain a single string in Go, it is possible for Go programmers to add additional information to errors by simply making the original error disappear. Version 1.13 passedfmt.ErrorfSupport for wrappers addresses this need.
  2. How to get the stack at the time of the error But in fact, in everyday development, there is another method that is used a lot, and that is to collect the stack at the time of the error. This absence of stack information is not friendly to Go novices. Consider two scenarios: 
    func Set(key string, value string) error{...return err 
}
Copy the code

    When a generic Set function returns an error, if its upper callers don’t handle it properly, the error is thrown up again, which can easily lead to an error that can’t be traced back to its source. The second case is in internal processing:

    func DoSomething(key string, value string) error{... err := io.Read()iferr ! =nil{
		return fmt.Errors("Read: %w",err)
	}
	...  
	err = io.Read()
	iferr ! =nil{
		return fmt.Errors("Read2: %w",err)
	}
}
Copy the code

    As you can see, there are multiple calls to a function (such as a Read call to Read a file). To distinguish err, we need to wrap each error, otherwise you won’t know which Read threw the problem. Still, it’s kind of gross to write code like that, and THERE’s nothing inherently good about Go. The open source community provides many error packages, such as:github.com/pkg/errors, is used to add errors to the stack, which is convenient to solve the above problems. Unfortunately, 1.13 does not include the stack feature, although it references the open source implementation.

  3. How to aggregate multiple errors In some special scenarios, such as in a sequence of processing logic, an error occurs, but you don’t want to interrupt the logic, you just want to record the error and continue. For example, there is a requirement that the statistics system needs to access the order, item, and user systems every night to pull statistics from each system. We might set up a timed task in the statistics system to pull every night, and when the pull fails, record an error and continue to pull the next one.
	func StartPull(a){
		var errMulti error
		for i := range systems{
			iferr := Pull(systems[i]); err ! =nil{
				errMulti = multierror.Append(errMulti, err)
			}
		}
	}
Copy the code

Here we use the github.com/hashicorp/go-multierror library, which is essentially a []error array. The important thing to note here is the difference between aggregating multiple errors and error wrappers. When aggregating errors, there can be no connection between errors. Error wrappers There is a hierarchy of errors.

5. Processing of the DAO layer

The division of C/S/D layer and the functions of each layer have been mentioned above. Here I want to talk about dao layer in detail.

1. Automatically generate code

In fact, if you write more DAO layer code, you will find that a lot of the code is generic, such as CURD code. It’s just a table change, the logic is the same. This code is very suitable for automatic generation using generators, such as: GORMT can automatically generate GORM CURD operations, used to write some small systems very easy.

2. Field hiding

It is not a good practice to write database table and field names in code. We usually use some structure, constants instead of writing directly in code. In this regard, you can also use tools to do automatic generation, no manual writing. For example: Gormt, Gopo, etc. With automatically generated field names and the powerful GORM V2 framework, our DAO layer can provide the functionality needed at the top level with minimal effort.

func FindBy(id int, columns ...string){
    db.Select(colums).Find(&SomeTable{},id)
}
// select only the ID Name field
bean := FindBy(1,dao.Colums.ID, dao.Colums.Name)
// Take only the Status field
bean := FindBy(2,dao.Colums.Status)
Copy the code

The sample code gives a simple example of a function called FindBy that allows upper-level calls to control which fields they want to retrieve without exposing the underlying database’s field names. The same can be done for Create, Update, etc., which I won’t repeat here.

6. The appendix

Although summarized 5 points, but I think, there are many small knowledge points, other aspects, may not be an article can write. If later encountered or encountered, will continue to sum up. After all, the ancients said, “Learning and learning, not also said.” Without a summary, you can’t see the whole picture. Some references are attached:

  • Shop a sample project I created is a complete running commodity service. Much of the sample code in this article is adapted from this project because it addresses some of the problems described above.
  • go-gin-apiThank youxinliangnoteAfter careful study of the project, I feel very inspired to me. A lot of code, modules, are learned from here.
  • GoFrame Although GoFrame is a bit too big, there is no denying that the code and documentation are very systematic and referential. It also has a detailed interpretation of the code structure.
  • Kratos Kratos framework in this article about Wire, relatively easy to understand. (As opposed to official wire documentation)
  • GoFrame link tracing is recommended, and link tracing is also good.