The original address

Github.com/anqiansong/…

github

github.com/anqiansong

Read the instructions

In this article, the table rendering of the URLs link is not very friendly. I suggest reading the original text

go module

With the release of GO 1.16, the Go Module has changed from the default value auto to ON, which means that the go Module mode is now recommended over the Gopath mode for future development.

In the past, I mostly developed Golang in the Go Module mode, but so far I am not familiar with it. I just stay in the following position: if others do this, I will follow them. This is not even the use of go Module, let alone familiarity or mastery. Until then, I have these questions:

• What the items defined in the Go mod file represent;
• In addition to the usualrequireOccasionally, seereplaceIn addition to keywords,exclude 、retract(1.16) What are these keywords and how to use them;
• What is the syntax format of the go mod file
• Github.com/tal-tech/go-zero v1.1.5 、Bcbc32de2 github.com/antlr/antlr4 v0.0.0-20210105212045-464 、Google.golang.org/protobuf v1.25.0 / / indirectWhat do the formats represent respectively, and why do they exist// indirectModification;
• Go. Mod has a go.sum (), and what does it do?
• .

I don’t know how many people like me know little about the Go Module.

Recently, with these doubts in mind, I went to study the official reference manual.

project

Before formally entering the module introduction, it is necessary to first understand the relationship between project and Module. I believe those who have developed Android or Java have a good understanding of module. Generally speaking, a project can be composed of multiple modules. Modules can be referred to as independent projects by other projects as dependencies. The following Golang project demo contains modules Foo and bar

├── ├─ ├─ ├─ goCopy the code

The module is introduced

Go Module (hereinafter called: Module, module, and project module) is a collection of packages of released versions in Golang, which is a way of Go to manage dependencies, similar to Gradle in Android and Maven in Java. Of course, their management forms are definitely different, but the purpose is the same, to manage dependencies.

In go.mod, it contains the Module path of the main Module, module dependencies, and associated information (version, etc.). If a project module needs to be developed in Go Module Mode (Module mode), it must contain the go.mod file in the root directory of the project module.

Module path(Module path)

The module path is the name of a project module, declared with the module command in go.mod, which is also the prefix of the package import in the project module. Let’s look at the module path under Demo /foo:

$ cat demo/foo/go.mod
Copy the code

Module github.com/foo go 1.16 require github.com/tal-tech/go-zeroCopy the code

github.com/foo is the module path of the main Module foo, github.com/tal-tech/go-zero is also the module path, they are also the prefix of package import in foo, I added an Echo function in the base package under foo and called it in main.go. Let’s look at the prefix of its package import

Directory tree

├── bass exercises ─ ├─ bass exercises ─ └Copy the code

main.go

package main

import "github.com/foo/base" // github.com/foo is module path
import "fmt"

func main(a) {
	msg := base.Echo("go-zero")
	fmt.Println(msg)
}
Copy the code

The main function of the Module path is to describe what an engineering module does and where to find it. Therefore, the component elements of the Module path include

• Repo path
• Repo folder
• version

Its manifestations are as follows: {{.rePO_URL}}/{{.nameofdir}}/{{.version}}, for example: github.com/tal-tech/go-zero/v2 and github.com/tal-tech explicitly tell the repo path, go-zero is the repo folder, v2 is the version number, and v1 is usually not written by default. Only if it’s greater than v1 does it need to be distinguished.

The version number

The version number here refers to the version of the dependent module in the go.mod file, which is related to {{.version}} above. V1.1.5 in the following example is the module dependent version number described in this article.

Module github.com/foo go 1.16 require github.com/tal-tech/go-zero v1.1.5Copy the code

Version number composition and rules

The version number consists of major Version, Minor Version, and Patch Version.

• Major version: indicates that backward incompatible changes have been made to the contents of the Module, and the version will be upgraded.minor version 和 patch versionIs to return to zero;
• Minor version: this version will be upgraded after new features are released or backward compatible content changes are made.patch versionIs to return to zero;
• Patch version: this version can be upgraded in case of bug repair or function optimization, or can be changed in case of pre-release release requirements

Examples: v0.0.0, v1.2.3, v1.2.10-pre

A version is considered unstable if its major version is 0 or patch-version has a version suffix (e.g. Pre). For example, v0.2.0, V1.5.0-pre, and V1.1.3-beta, you can refer to Semantic Versioning 2.0.0 for more information about version semantics.

Golang also uses tags and branches to represent a version, such as: 39540 e21d249e91f89d96d015a6e3795cfb2be44 github.com/tal-tech/go-zero github.com/tal-tech/go-zero E21d249, github.com/tal-tech/go-zero@master v1.1.6-0.20210303091609-39540

github.com/tal-tech/go-zero v1.1.6-0.20210303091609-39540e21D249 is called pseudo-versions in Golang. It does not completely follow the version rules above, and the pseudo-version consists of three parts:

• Version base prefix (vX.Y.Z-0 或 VX. 0.0), such asV1.1.6 0
• Timestamp: The creation time of revision, as in20210303091609
• Revision identifiers, such as39540e21d249

Pseudo versions can take three forms, depending on the base prefix:

• VX. 0.0 - yyyymmddhhmmss - abcdefabcdef: in the absence ofreleaseVersion time
• vX.Y.Z-pre.0.yyyymmddhhmmss-abcdefabcdef: Used when the base version is a pre-release version
• vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdefabcdef: whenreleaseVersion of the similarvX.Y.ZWhen, as inE21d249 github.com/tal-tech/go-zero v1.1.6-0.20210303091609-39540 的 releaseVersion forv1.1.5

Pseudo versions do not need to be entered manually, but are automatically converted to pseudo versions when partial go commands are executed to retrieve code from a certain revision version as a dependency

github.com/tal-tech/go-zero v1.1.6-0.20210303091609-39540e21d249 in the above is executing go get github.com/tal-tech/go-zero After 39540 e21d249e91f89d96d015a6e3795cfb2be44 automatic conversion results

Version of the suffix

For forward compatibility, if major version is upgraded to 2, the module path must specify a version suffix v2 (the value must be the same as the major version value in the version). If major version is less than 2, the version suffix is not allowed.

For example, I used the miniRedis library in the Demo /foo module project. The major version of the miniRedis library has been upgraded to 2. What if I referenced the following version in go.mod?

The module github.com/foo go 1.16 / / into / / the require correct github.com/alicebob/miniredis/v2 v2.14.1 / / error introduced the require Github.com/alicebob/miniredis v2.14.1Copy the code

invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2
Copy the code

Let’s look at what happens when the main Module release is upgraded to v2.x.x and the Module path extension is not added to another module: Example Module foo The Foo project currently has a release

• v1.0.0
• The v2.0.1
• .

Use module’s project bar

• Before the version suffix is added

Foo project directory tree

├── └. Go ├─ goCopy the code

The module path for github.com/anqiansong/foo

The module github.com/anqiansong/foo go 1.16Copy the code

Use v2.0.0 version of go.mod in project bar

The require github.com/anqiansong/foo v2.0.0Copy the code

You will find that the error message is

The require github.com/anqiansong/foo: reading https://goproxy.cn/github.com/anqiansong/foo/@v/v2.0.0.info: 404 Not Found server response: Not Found: github.com/anqiansong/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2Copy the code

If the require github.com/anqiansong/foo v1.0.0 is possible.

• After the version suffix is added

Foo project directory tree

├── └. Go ├─ goCopy the code

The module path for github.com/anqiansong/foo/v2

The module github.com/anqiansong/foo/v2 go 1.16Copy the code

Use v2.0.1 version of go.mod in project Bar

The require github.com/anqiansong/foo/v2 v2.0.1Copy the code

Bar runs normally

If the major version is upgraded to V2 and the version is not intended for forward compatibility and does not want to add the version suffix to the Module path, the build tag can be terminated with +incompatible. The other project reference sample for the require github.com/anqiansong/foo v2.0.0 + incompatible

How to parse module in package

The Go command first searches the build list for modules with a package path prefix. For example, if the package example.com/a/b is imported and the module example.com/a is in the build list, the go command will check if example.com/a contains packages in directory B. And the directory contains at least one GO file to be considered package. Build constraints should not be used for this purpose. If only one module in the generated list provides packages, use that module. If no module provides a package, or if two or more modules provide a package, the go command reports an error. The mod=mod flag instructs the go command to try to find the new module that provides the missing package and to update go.mod and go.sum. The go get and Go mod Tidy commands do this automatically.

When the go command updates or retrieves module dependencies, it checks the GOPROXY environment variable. The value of GOPROXY is a comma-separated list of urls, or the keywords direct, off,

• The comma-separated url is the proxy address, which is toldgoCommand to initiate a connection with this value
• direct: Specifies module dependencies to be obtained through version control
• off: does not attempt to connect to the module

If GOPROXY sets a specific URL, suppose the go command is looking for a package of github.com/tal-tech/go-zero/zrpc, the go command will look for the Module in parallel

• github.com/tal-tech/go-zero/zrpc
• github.com/tal-tech/go-zero
• github.com/tal-tech
• github.com

If one or more of them match to contain content that satisfies github.com/tal-tech/go-zero/zrpc, take the longest module as a dependency. After finding the appropriate module and version, The go command adds require to the go.mod and go.sum files. If the module is not imported by the main Module, add // direct to the value of require. An error; If GOPROXY has multiple URL proxies, the steps above are performed for subsequent proxies in the event of previous failures.

Go. The mod file

A module is identified by having a utF-8 encoded text file named go.mod in its root directory. The contents of the go.mod file are line-oriented, each line contains an instruction, and each line consists of a keyword and a parameter, as in:

The module github.com/anqiansong/foo go 1.16 the require github.com/tal-tech/go-zero require github.com/tal-tech/go-queue Replace go.etcd. IO /etcd => go.etcd. IO /etcd v0.0.0-20200402134248-51bdeb39e698 retract [v1.0.0, v1.0.1] replace go.etcd. IO /etcd v0.0.0-20200402134248-51bdeb39e698 retract [v1.0.0, v1.0.1]Copy the code

Of course, content with the same keyword can be separated out and used as a keyword + block, as in:

Replace the module github.com/anqiansong/foo go 1.16 the require (github.com/tal-tech/go-zero github.com/tal-tech/go-queue) IO /etcd => go.etcd. IO /etcd v0.0.0-20200402134248-51bdeb39e698 retract [v1.0.0, v1.0.1]Copy the code

Go. mod is machine-writable, and executing commands such as Go get and Go mod Edit may automatically update the go.mod file.

Module component

When parsing the contents of the go.mod file, it will be parsed as

• Whitespace characters: Contains space (U+0020), TAB (U+0009), carriage return (U+000D), and line feed (U+000A).
• annotation: Comment Only single-line comments are supported//
• punctuation: Punctuation does( 、 ) 、 . 、 = >
• The keyword ： go 、 require 、 replace 、 exclude 、 retract
• identifier: by theWhitespace charactersA sequence of characters, such as Module path, semantic version
• string: Contains double quotation marks"(U+0022) The explanation string for the package is either available<(U+0060) the original string wrapped. Such as"github/com/tal-tech/go-zero", ` `

Identifiers and strings are interchangeable in the go.mod syntax

Module syntax and morphology

The go.mod syntax is defined via Extended Backus-naur Form (EBNF paradigm), as in

GoMod = { Directive } .
Directive = ModuleDirective |
            GoDirective |
            RequireDirective |
            ExcludeDirective |
            ReplaceDirective |
            RetractDirective .
Copy the code

moduleinstruction

The module keyword defines the module path of the main Module, with only one module specified in the go.mod file.

Grammar rules:

ModuleDirective = "module" ( ModulePath | "(" newline ModulePath newline ")" newline .
Copy the code

Example:

module github.com/tal-tech/go-zero
Copy the code

goinstruction

The go keyword defines the version of go expected to be used by the Module Settings. The version must be a valid version of Go.

With the go keyword, the compiler knows when the package is compiled which go version should be used. In addition, the go keyword definition version can also be used to enable some of the go command features, such as whether to automatically start vendoring at version 1.14 and later.

Grammar rules:

GoDirective = "go" GoVersion newline .
GoVersion = string | ident .  /* valid release version; see above */
Copy the code

Example:

Go to 1.16Copy the code

requireinstruction

Require declares the minimum version of a Module dependency. After require specifies the version, go will load the dependency based on this value according to MVS rules.

When GO finds a dependency that is not directly dependent on the Main Module, it adds a // direct comment to the module path.

Grammar rules:

RequireDirective = "require" ( RequireSpec | "(" newline { RequireSpec } ")" newline ) .
RequireSpec = ModulePath Version newline .
Copy the code

Example:

Module github.com/tal-tech/go-zero go 1.16 require (golang.org/x/crypto v1.4.5 // Indirect golang.org/x/text v1.6.7)Copy the code

exculeinstruction

Excule ignores specified versions of content. From Go 1.16, modules specified by exclude are ignored, and before Go 1.16, if require modules specified by exclude, Lists and retrieves the changed versions as excluded.

Grammar rules:

ExcludeDirective = "exclude" ( ExcludeSpec | "(" newline { ExcludeSpec } ")" ) .
ExcludeSpec = ModulePath Version newline .
Copy the code

Example:

Module github.com/tal-tech/go-zero go 1.16 Exclude golang.org/x/net v1.2.3 excule (golang.org/x/crypto v1.4.5 Golang.org/x/text v1.6.7)Copy the code

replaceinstruction

The replace directive is used to replace the specified version of a module or module with another module or version. If the version is specified on the left, replace that version with the target content; otherwise, replace all versions of a Module with the target content

Grammar rules:

ReplaceDirective = "replace" ( ReplaceSpec | "(" newline { ReplaceSpec } ")" newline ")" ) .
ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline
            | ModulePath [ Version ] "=>" ModulePath Version newline .
FilePath = /* platform-specific relative or absolute file path */
Copy the code

Example:

Replace golang.org/x/net v1.2.3 => example.com/fork/net v1.4.5Copy the code

=> The content on the right can be a valid Module path or a relative or absolute path. If it is a relative or absolute path, the root of the path must contain the go.mod file.

Example:

Require github.com/foo v1.0.0 replace github.com/foo v1.0.0 =>.. /barCopy the code

retractDirective (Added in 1.16)

Retract: Retract: retract: Retract: Retract: Retract: Retract: Retract For retract, a warning will be displayed. Go list-m-versions will also hide the version.

Grammar rules:

RetractDirective = "retract" ( RetractSpec | "(" newline { RetractSpec } ")" ) .
RetractSpec = ( Version | "[" Version "," Version "]" ) newline .
Copy the code

Example:

// someting wrong retract v1.0.0 // Someting wrong in range of versions => v1.0.0 to v1.2.0 retrace [v1.0.0,v1.2.0]Copy the code

Let’s look at an example, the current github.com/anqiansong/retract already has v1.0.0 etc version, we add a line retract instruction tag v1.0.0 withdrawal:

// someting wrong
retract v1.0.0
Copy the code

Then release a version for v1.0.1, the following quote v1.0.0 in github.com/anqiansong/bar version

The require github.com/anqiansong/retract v1.0.0Copy the code

Then go go get github.com/anqiansong/[email protected], not surprisingly, will get a prompt contains something wrong and prompt update to v1.0.1 information

$Go get github.com/anqiansong/[email protected]
Copy the code

Go: warning: github.com/anqiansong/[email protected]: retracted by the module author: someting wrong go: to switch to the latest unretracted version, run: go get github.com/anqiansong/retract@latestgo get: Downgraded github.com/anqiansong/retract v1.0.1 = > v1.0.0Copy the code

Get github.com/anqiansong/retract all module release version

$ go list -m -versions github.com/anqiansong/retract
Copy the code

Github.com/anqiansong/retract v1.0.1Copy the code

Part of the command to see github.com/anqiansong/[email protected] results:

• go get

  $Go get github.com/anqiansong/[email protected]
  Copy the code

  Go: warning: github.com/anqiansong/[email protected]: retracted by the module author: someting wrong go: to switch to the latest unretracted version, run: go get github.com/anqiansong/retract@latestCopy the code

• go list -m -u

  $Go get github.com/anqiansong/[email protected]
  Copy the code

  Github.com/anqiansong/retract v1.0.0 (retracted) [v1.0.1]Copy the code

• go list -m -versions

    $ go list -m -versions github.com/anqiansong/retract
  Copy the code

  Github.com/anqiansong/retract v1.0.1Copy the code

Description:

Retract controls the main Module version, not the dependent module version.

Retract can be referenced by other modules, but some of the go commands have different retract results, as shown above.

Automatic updates

Most go commands will report errors if go.mod lacks information or does not accurately reflect reality. The go get, go mod Tidy commands can be used to fix most of these problems. In addition, the -mod=mod flag can be used with most mode-aware commands (go build, go test, etc.) to indicate that the go command automatically fixes problems in go.mod and go.sum.

The module perception

Most go commands can be run in module-ware or GOPATH mode. In module-ware mode, the go command uses the go.mod file to look for version dependencies, which typically loads packages from the Module cache and downloads modules if they are missing. In GOPATH mode, the go command ignores module; It looks for dependencies in the Vendor or GOPATH directory.

In Go 1.16, module-ware mode is enabled by default, regardless of whether a go.mod file exists. In earlier versions, module-ware mode is enabled when a go.mod file exists in the working directory file or any parent directory.

Module-ware mode can be controlled by the GO111MODULE environment variable, which can be set to ON, OFF, or Auto

• off ：goRelated commands are ignoredgo.modFile, and then toGOPATHmode
• on ： onOr an empty stringModule-waremode
• auto: If the current folder existsgo.modFile, will be withModule-wareMode running on Go 1.15 and later, this value is the default,

Some go Module commands

The command must be in module-ware mode to be valid

Proxy

The module proxy is an HTTP server that supports a response to a GET request, which has no query parameter and does not even require specific header information, even if the value is a fixed file system site (e.g., file:// URL).

The module proxy’s HTTP response status code must contain 200 (OK), 3xx, 4xx, 5xx, 4xx, 5XX as an error response, 404 and 410 indicate that all module requests are unavailable. The contentType of the error response should be set to TEXT /plain with utF-8 or US-ASCII character set.

URLs

The go command can be read by GOPROXY environment variable configuration to connect to connect a proxy server or the version control system, GOPROXY accept a comma (,) or a vertical bar (|) integral multiple url value, when the comma (,) in English, Only when the response status code is 404 or 410 will try to the back of the proxy address, if is a vertical bar (|) segmentation, any mistakes (including overtime) in the HTTP will skip to try the back of the agent address. It can also be the direct or off keyword.

The following table shows the path that a proxy address must implement and have a request response (i.e. a proxy server must support the implementation of the following routes)

• $baseIs the proxy server address, for example:goproxy.cn
• $moduleModule path, for example, github.com/tal-tech/go-zero
• $versionFor the module version

  type Info struct {
      Version string    // version string
      Time    time.Time // commit time
  }
  

{" Version ":" v1.1.5 ", "Time" : "the 2021-03-02 T03:02:57 Z}"

module github.com/tal-tech/go-zero
Go to 1.14
Require (github.com/ClickHouse/clickhouse-go v1.4.3 github.com/DATA-DOG/go-sqlmock v1.4.1 Github.com/alicebob/miniredis/v2 v2.14.1 github.com/antlr/antlr4 v0.0.0-20210105212045-464 bcbc32de2...).

{" Version ":" v1.1.5 ", "Time" : "the 2021-03-02 T03:02:57 Z}"

$base/$module/@v/list = latest $base/$module/@v/list = latest $base/$module/@latest Go commands are sorted by release, pre-release, and pseudo.

The $base/$module/$version.mod and $base/$module/$version.zip addresses must be provided because they can be used to verify data with go.sum.

Go after download the module content usually stored in $GOPATH/PKG/mod/cache/download path, including the download version control system.

direct

If GOPROXY is set to direct, the Module resources will be downloaded from the version control system (git, SVN, etc.) when the go command is executed. Of course, the other two environment variables GOPRIVATE and GONOPROXY are also required. For more information about environment variables, see here

Module file size constraints

There are many restrictions on the contents of the module ZIP file. These constraints ensure that compressed files can be extracted safely and consistently across a wide range of platforms.

Up to 500MiB per module (either compressed or uncompressed files)

The go.mod file must be displayed within 16MiB

go.sum

There may be a file named go.sum at the root of the module that is attached directly to go.mod, and when the go command is executed to get the module, it checks to see if the zip file and go.sum match.

The value in go.sum consists of a Module path, version, and a hash value, such as:

Github.com/anqiansong/retract v1.0.1 h1: jxcsUM / 6 tvxm7p14 / XMeZPFbql5KAAZJfFqiHG + YKxA =Copy the code

conclusion

Spent 3 days, the first day watch the original document in English and translate into Chinese to verify the second and third day own understanding, and write this post, after learning, to go the module handbook learning to solve a lot of doubt, at least a few opening problem is solved, the paper is also part of the manual handling combined with some of his own understanding, And with real example to validate, which still has a lot of content I didn’t completely copy, if students are interested in, you can refer to the official manual, part contains personal understanding, this paper is combined with Google translation plus for some unreasonable places human translation, certainly exist unreasonable place, if you have to understand is not consistent, Suggestions and discussions are welcome.

Reference documentation

• Semantic Versioning 2.0.0
• Go Module Official Documentation