• Go package management has had a rough ride, but by the time I started learning Go, Go Modules were already widely used. Therefore, I did not experience go DEP, Go Vendor, etc., and directly went from the GOPATH mode used in the learning stage to the Go Modules at work.
  • When you first started using Go Modules, the two commands Go mod init + Go mod Tidy seemed to be enough, and the concepts involved were vague.
  • There are a lot of tools that are as important and transparent as air in a development project, and you don’t need to know too much about them. A few basic commands, such as Git and Go Modules, can solve most of the problems. Their design is very clever when it takes a little time to learn them carefully. This article intends to record some knowledge points about GOPATH and Go Modules to deepen my impression of them.

GOPATH mode

There are three important environment variables for installing the GO language:

GOROOT GOBIN GOPATH
Path to install the Go language Path to save the executable file generated by the Go program The workspace

GOROOT is a path to install the GO language, nothing special.

The key is GOPATH, whose configuration value is a directory. Multiple directories can be configured, and each directory serves as a workspace for the GO language. Without Gomod enabled, go code can only exist under GOPATH, which has its own unique code organization.

A few important commands

  • Before we look at how GOPATH code is organized, let’s examine a few important commands, which are:
    • go build
    • go install
    • go get

go build

  • Building code
  • If you build code with the main method, you get an executable file that appears in the same directory as the source file.
  • If code is executed without main, it must be called directly or indirectly by main. The executable calls the imported code through the link library file. To build code without the main method, you get link library files with a “. A “extension, which is the archive of link library files. The archive built from the same source file varies from operating system to operating system.
  • The built archive files are stored in a temporary directory.

go install

  • Installation code
  • Before installing the code, the code is first built, that is, go install = go Build + other actions.
  • The other operations here refer to link operations
    • The result of a code build with the main method is moved to the GOBIN directory or, if GOBIN is not configured, to the bin directory of the GOPATH where it is located.
    • Code builds without the main method are moved to the PKG directory under GOPATH.

go get

  • Download the code
  • Automatically pull from a mainstream public repository, such as Github, and execute go Install.
  • It can be seen that these three commands are “stacked”.

GOPATH

  • GOPATH can be configured with many.
  • GOPATH/bin and GOPATH/ PKG directories are mentioned in the analysis of the commands, which fall under the category of how GOPATH organizes code.
  • Within each GOPATH directory, there are three subdirectories
    • src
      • Used to store source code. When writing your own code, the parent directory of the code path of import is SRC under a GOPATH.
    • pkg
    • bin

Use the go get github.com/gin-gonic/gin result to illustrate how the GO code is organized.

Note: This can only be done if the Go Modules are turned off, as GO111MODULE=off in Go env. The situation under Go Modules is analyzed in the Go Modules section.

src

  • The SRC directory holds the code
  • The code is stored at SRC /github.com/gin-gonic/gin, and the directory level follows the go get path (split by /)

pkg

  • Archive files generated during the build of Go Install and Go Get are placed here, at the directory level and below SRC.
  • As mentioned earlier, there is an extra layer of operating system platform directory than SRC directory. For example, the gin file level under Linux PKG is PKG /linux_amd64/github.com/gin-gonic/gin

bin

  • This is where the executable files generated during the build of Go Install and Go Get will be if GOBIN is not configured.

GOPATH’s most fatal flaw

  • Package versioning is inconvenient.

  • If project A uses GIN version V1 and Project B uses GIN version V2.

  • Then you can only create two GOPATH for project A and project B, and put the required version of gin code under your own GOPATH.

Go Modules

  • Go Modules came out in version 1.11, so the switch configuration for Go Modules is called GO111MODULE. After version 1.13, Go Modules is set as the default package management tool for Go. For its value, I think brainless on will do. Of course, in addition to off, it also has an auto option, which I personally think is not very meaningful.

Leading-edge concepts: Package and Module

  • Package is the go code package. The first line of the GO code declares the code package to which the change code belongs. Code packages can be divided into:

    • Go standard package
    • Third-party packages, such as those from Github.com
    • The anonymous package, import _ ***, mainly performs the init function of the imported package
    • Internal packages, other code packages in the same project

  • A module is a collection of packages

  • In Go Modules, you can think of each project as a module with a go.mod file in its root directory. It defines the name of this module and other modules that it depends on, and its meaning will be explained later.

Go mod command

  • A few key commands are briefly listed, and their usage and logic will be shown later.
The command role
init Initialize the current directory to a new module
download Download all dependent packages
graph View the dependency structure
tidy Add missing modules (imported in project, not in gomod file), remove useless modules (opposite)
vendor Save all dependency packages in the vendor directory under the current project.
verify Check that the current module’s dependencies are already stored in the locally downloaded source code cache and that they have been modified since the download.
why See why you need to rely on a module

Download the Go Modules

  • After the Go Module is enabled, both the steps and the results of the Go Get execution are different from those in GOPATH mode. This section focuses on the go Get process and the results will be discussed later.

Download source changes

  • An environment variable is added to the Go Module :GOPROXY. Used to set the Go module agent (download agent).
  • After GOPROXY is configured, the go Get command will first download the code from GOPROXY. This can solve many problems, such as:
    • If the repository is walled or very slow to access, GOPROXY can avoid the problem by setting the address to be accessible in the country. Goproxy.cn provided by Qiniu is generally used.
    • Some code in the repository may be deleted, and the proxy server stores it permanently, addressing the risk of loss.
    • Some companies can also build their own proxy servers, which are faster.
  • GOPROXY can be configured with multiple proxy servers separated by commas (,). It will be downloaded in order.
  • In addition, GOPROXY can be configured with a special value: direct, which indicates the return source. Go directly to the real path after go get. If GOPROXY is set to goproxy.cn,direct go get github.com/gin-gonic/gin will download the code from goproxy.cn first. If this fails, go back to github.com/gin-gonic/gin to download the code.
  • There are also two related environment configurations:GONOPROXYandGONOSUMDB.
    • GONOPROXY is for GOPROXY. There may be some private repositories that cannot be obtained from proxy servers during project development, such as code on gitLab within the company, which requires GONOPROXY configuration. For example, if GONOPROXY is configured as the company’s GitLab address, when pulling code from the company’s GitLab, GOPROXY will not be pulled.
    • GONOPROXY will not be able to check the code pulled back from GONOPROXY, and GONOSUMDB will be configured to bypass the check. The logic of the inspection is followed by a detailed analysis.
    • Obviously GONOPROXY and GONOSUMDB are generally the same, and there are environment configurations of GOPRIVATE that can be used as their default values, so you just need to configure GOPRIVATE.

Download different versions

  • Modules can also be downloaded by specifying the version at go get package@verison. For example, download v1.7.6 of GIN. Go get github.com/gin-gonic/[email protected].
  • It is common in development to encounter a module that depends on many other modules, and different modules depend on different versions of the same module.
    • As an example of how to deal with Go Modules, the following letters represent a module, and -> represents dependencies.
    • Main body of the dependencies: A – > B, C – > D | | B C – > D, the F | D – > E. Each module has a different code version depending on the version number.
    • The Go Modules select the version like this:
    • There are three dependent links starting from A:
      • A->B->D->E
      • A->C->D->E
      • A->C->F
    • Among the three dependent links, the one that may have version differences is D. Because both B and C depend on D, it’s possible to have a dependency on different versions of D.
    • At this point, the Go Modules selects the higher version of D. because
      • If it is a minor version difference, such as V1.1.0 and V1.2.0. According to the specification, the higher version should be compatible with the lower version.
      • If it is a big version difference, such as V1.0.0 and V2.0.0. Their module import paths should be different! Insert an eye here, and we’ll refer to this again when we describe the Gomod file.

Go. The mod file

  • In this file, you define the name of the module and the packages it depends on, and each dependency package needs to specify the import path and semantic version to be accurately described.
1.17 the require module gitlab.xxx.com/hello/gomodules go (github.com/dgrijalva/jwt-go v3.2.0 + incompatible github.com/gogf/gf v1.16.6 github.com/go-sql-driver/mysql v1.6.0 // Indirect gitlab.com/wang/utils v1.1.0 Gopkg. in/yaml.v3 v3.0.0-20210107192923-496545a6307b // indirect) replace (gitlab.com/wang/utils => /home/wang/utils) Exclude (github.com/gogf/guuid v1.0.0)Copy the code
  • This is a random GOMod file, the purpose of which is to cover multiple keywords to analyze their meaning
  • List the keywords that appear first:
    • module
    • go
    • require
    • replace
    • exclude
    • vx.x.x
    • +incompatible
    • indirect

module

  • Declare the name of this module. Is also the module path of this module.

go

  • Declaring the GO version of this module is just a declaration and has no practical use.

When creating a new project, go Modules will be initialized with gomod init XXX. This command creates a gomod file with XXX after init as the module name and the go version of the development environment as the GO version in the gomod file.

require

  • Set specific dependency modules in the format [module path] [version]
  • Version details are described in vx.x.

When the external module code needs to be introduced during the development of the project, it is usually necessary to import the path of the external module in the code first, and then use the Go mod Tidy command. At this time, the code will be automatically downloaded and the gomod file will be updated, but the version cannot be specified. If you need to specify the version, write the module path and version in the require section of the gomod file and then run the gomod Tidy command.

replace

  • Replace the import path specified in require with the format [module] => [newModule]
  • So when you download the package, you go to NewModule to download it.
  • The newModule can be a network address or a local path. Replace is mainly used to solve problems that cannot be pulled, such as network failure or download permissions.

exclude

  • Used to exclude a module.

Note: Replace and exclude are valid for the current module and do not affect other modules that the current module depends on.

vx.x.x

  • The version number
  • If the dependent module has a tag in the semantic version format, the value of the tag is displayed directly. Such as github.com/gogf/gf v1.16.6. V1.16.6 is the version number.
  • In /yaml. V3 v3.0.0-20210107192922-496545a6307b. In /yaml. V3 v3.0.0-20210107192922-496545a6307b

+incompatible

  • If a larger version of a module is v0 or v1, there are no special requirements for its module path, such as github.com/gogf/gf v1.16.6.
  • However, if the major version number is greater than v1, then the major version must appear at the end of the module path, such as github.com/appleboy/gin-jwt/v2 v2.6.3. If not, +incompatible will be automatically added to the version number when we go Mod Tidy, such as github.com/dgrijalva/jwt-go v3.2.0+incompatible. TP to Download the different versions in the download to Go Modules section.

indirect

  • Represents a module that is not directly dependent. There are two reasons for indirect dependencies:

    • When Go Modules are not enabled, for example, A depends on B, which in turn depends on B1 and B2. But B does not have the go.mod file, so B1 and B2 will be recorded in A’s go.mod,

      Add // indirect.

    • There are some flaws in the go.mod file where A depends on B, which in turn depends on B1 and B2. B has A go.mod file, but only B1 is recorded in B’s go.mod file. In this case, B2 is recorded in A’s go.mod file with // indirect.

  • In my own development environment, go version 1.17, I found that when GO Mod Tidy, I would separate the indirect dependency from the direct dependency in another require section.

Go. Sum file

  • This file is one of the most nonexistent files I’ve used in my early days, but a closer look at it shows how useful it is.

  • Before we get into it, let’s explain where the Go Get code goes when we turn on the Go Modules. For example, run the go get github.com/gin-gonic/[email protected] command.

    • Perform the go after the get command, $GOPATH/pkg/mod/cache/download/github.com/gin-gonic/gin/@v will appear under several files:
      • V1.7.6. zip: compressed package of code
      • V1.7.6. Ziphash: Hash value of the code
      • V1.7.6. mod: gomod file of the module
      • V1.7.6. info: basic module information
    • In addition, will create $GOPATH/pkg/mod/github.com/gin-gonic/[email protected] folder, save the code.
    • From the code store catalog described in the appeal process, it is clear that multiple versions can be saved.

  • If you think about it, what are the problems with current Go Modules?

    • When you just pulled a project back with Git, you pulled back the project code and the go.mod file. Then execute Go Mod Tidy to download the other modules that the project depends on. But what if the source code that comes back is tampered with, that is, not the same code that the project developers were relying on at the time? This can lead to problems with the project.
    • Or the code already exists under $GOPATH/ PKG /mod, but can be changed. How to deal with it?
      • Although the entire $GOPATH/ PKG /mod is set to read-only mode, there is no way to completely avoid tampering.

  • Go. Sum is there to solve problems.

  • Here is the go.sum file for the dependent module:

    Github.com/BurntSushi/toml v0.3.1 h1: WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ = github.com/BurntSushi/toml V0.3.1 / go mod h1: xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU =Copy the code
    • A dependency module typically has two lines:
      • First line :[module] [version] [Hash algorithm] :[hash]. Just note that h1 refers to the hash algorithm used, SHA-256.
      • Line 2 :[module] [version/go.mod] [Hash algorithm] :[hash]. The first line hashes the code and the second line hashes the gomod file. Of course this module does not have a gomod file, there is no second line.

  • How is the content in go.sum generated?

    • When downloading dependencies from the project root directory using the go get command, the go.mod and go.sum files are updated in addition to the above results.
    • There is another key step from the Go Get download to updating the go.sum file, which involves an environment configurationGOSUMDB.
      • The value of GOSUMDB is a Web server, which defaults to sum.golang.org. The service can query the hash value of the version specified by the dependent package. In other words, for example, go Get Gin’s V1.7.6 code will calculate the hash value of the code and compare it with the query result on Sum.golang.org. If the result is different, the go Get code will be considered tampered with.
      • Of course, the GOSUMDB server can be built by itself. It can also be set to off, which of course is risky.
      • So, when go.sum is updated, it is considered checked. Its hash value is trusted.
    • Through this operation, the hash value in go.sum is used as a baseline for comparison.

  • This solves the Go Modules problem. There are two cases of the go get operation in the project directory:

    • If the code is not local, download it first, and then check whether it is tampered with by GOSUMDB. If the check is passed, cache it locally
      • Go. Sum is entered if no module information exists.
      • If it exists, make a comparison and report an error if it is inconsistent. (The existing Go.sum is the benchmark for the test)
    • If the module code is already local:
      • Go. Sum does not contain module information. (GOSUMDB has been checked during the download process, but the local cache is read-only, so it is considered reliable and can be directly written to go.sum)
      • If it exists, make a comparison and report an error if it is inconsistent.

Internal package

  • If some code is not used by other modules, you can create an internal code package in your own module, and the code in the package is not referenced by other modules.